Loss in localization accuracy was tackled by pioneering works in Deep Semantic Segmentation, such as SegNet #REFR .

The exploration of larger (ultimately global) contexts, in turn, can be performed by the stacking of convolutional layers so that the receptive fields get larger at each level, as illustrated in Fig 3.

Depending on the input resolution, though, achieving a large receptive field can result in very deep networks, which incur high computational costs, and can lead to problems such as vanishing or exploding gradients, as well as overfitting issues.

Given this context, some alternatives are widely adopted in the literature for a good balance of local and global feature exploration.

1) Fast Downsampling: The first alternative is to perform a fast down-sampling of feature maps, in order to fully extract contextual information with fewer layers.However, this leads to some of the main challenges to highly-accurate semantic segmentation, as stated by Atif et al.

#OTHEREFR : reduced localization accuracy introduced by pooling operations to incorporate invariance in the network, and reduced feature map resolution caused by strided convolutions.

They proposed to keep spatial details by preserving the location of the values extracted by the max pooling operation in the encoder stages.

This information would then be passed to the corresponding decoder layer by means of the so-called Pooling Indices in order to guide the up-sampling stages. #OTHEREFR and #OTHEREFR try a similar approach.

Excessive downsampling of feature maps leads to the loss of information related to small objects.

In FCNs, for instance, the output is usually 32 times smaller than the input image #OTHEREFR .

Throughout the years, various approaches have been proposed in order to reduce loss of spatial details caused by feature downsampling, while guaranteeing the coverage of long-range dependencies.

A Threefold Review On Deep Semantic Segmentation: Efficiency-Oriented, Temporal And Depth-Aware Design

Abstract: Semantic image and video segmentation stand among the most important tasks in computer vision nowadays, since they provide a complete and meaningful representation of the environment by means of a dense classification of the pixels in a given scene. Recently, Deep Learning, and more precisely Convolutional Neural Networks, have boosted semantic segmentation to a new level in terms of performance and generalization capabilities. However, designing Deep Semantic Segmentation models is a complex task, as it may involve application-dependent aspects. Particularly, when considering autonomous driving applications, the robustness-efficiency tradeoff, as well as intrinsic limitations-computational/memory bounds and data-scarcity-and constraints-real-time inference-should be taken into consideration. In this respect, the use of additional data modalities, such as depth perception for reasoning on the geometry of a scene, and temporal cues from videos to explore redundancy and consistency, are promising directions yet not explored to their full potential in the literature. In this paper, we conduct a survey on the most relevant and recent advances in Deep Semantic Segmentation in the context of vision for autonomous vehicles, from three different perspectives: efficiency-oriented model development for real-time operation, RGB-Depth data integration (RGB-D semantic segmentation), and the use of temporal information from videos in temporalaware models. Our main objective is to provide a comprehensive discussion on the main methods, advantages, limitations, results and challenges faced from each perspective, so that the reader can not only get started, but also be up to date in respect to recent advances in this exciting and challenging research field.

Segnet: A Deep Convolutional Encoder-Decoder Architecture For Image Segmentation

Abstract: We present a novel and practical deep fully convolutional neural network architecture for semantic pixel-wise segmentation termed SegNet. This core trainable segmentation engine consists of an encoder network, a corresponding decoder network followed by a pixel-wise classification layer. The architecture of the encoder network is topologically identical to the 13 convolutional layers in the VGG16 network [1] . The role of the decoder network is to map the low resolution encoder feature maps to full input resolution feature maps for pixel-wise classification. The novelty of SegNet lies is in the manner in which the decoder upsamples its lower resolution input feature map(s). Specifically, the decoder uses pooling indices computed in the max-pooling step of the corresponding encoder to perform non-linear upsampling. This eliminates the need for learning to upsample. The upsampled maps are sparse and are then convolved with trainable filters to produce dense feature maps. We compare our proposed architecture with the widely adopted FCN [2] and also with the well known DeepLab-LargeFOV [3] , DeconvNet [4] architectures. This comparison reveals the memory versus accuracy trade-off involved in achieving good segmentation performance. SegNet was primarily motivated by scene understanding applications. Hence, it is designed to be efficient both in terms of memory and computational time during inference. It is also significantly smaller in the number of trainable parameters than other competing architectures and can be trained end-to-end using stochastic gradient descent. We also performed a controlled benchmark of SegNet and other architectures on both road scenes and SUN RGB-D indoor scene segmentation tasks. These quantitative assessments show that SegNet provides good performance with competitive inference time and most efficient inference memory-wise as compared to other architectures. We also provide a Caffe implementation of SegNet and a web demo at http://mi.eng.cam.ac.uk/projects/segnet.


We also have tested the proposed method in MP3D #REFR dataset, and observed similar results with Gibson dataset.

F loc , the success rate and SPL dramatically drop.

We provide the results in the supplementary material E, and additional ablation study about the submodules of the navigation module is also provided in D.

Comparing the third row and the last row of Table 2 , we can infer that the pose adjusting function F track helps the agent to find the image goal more effectively, even with the noisy odometry.

The proposed method shows higher success rates in noisy settings, while SPL values are low.

We believe this is from the randomness of noises, which helps the local navigation policy to escape from being stuck in a single location.

Renderable Neural Radiance Map For Visual Navigation

Abstract: We propose a novel type of map for visual navigation, a renderable neural radiance map (RNR-Map), which is designed to contain the overall visual information of a 3D environment. The RNR-Map has a grid form and consists of latent codes at each pixel. These latent codes are embedded from image observations, and can be converted to the neural radiance field which enables image rendering given a camera pose. The recorded latent codes implicitly contain visual information about the environment, which makes the RNR-Map visually descriptive. This visual information in RNR-Map can be a useful guideline for visual localization and navigation. We develop localization and navigation frameworks that can effectively utilize the RNR-Map. We evaluate the proposed frameworks on camera tracking, visual localization, and image-goal navigation. Experimental results show that the RNR-Map-based localization framework can find the target location based on a single query image with fast speed and competitive accuracy compared to other baselines. Also, this localization framework is robust to environmental changes, and even finds the most visually similar places when a query image from a different environment is given. The proposed navigation framework outperforms the existing image-goal navigation methods in difficult scenarios, under odometry and actuation noises. The navigation framework shows 65.7% success rate in curved scenarios of the NRNS [23] dataset, which is an improvement of 18.6% over the current state-of-the-art. Project page: https: //rllab-snu.github.io/projects/RNR-Map/

Matterport3D: Learning From Rgb-D Data In Indoor Environments

Abstract: Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms. However, existing datasets still cover only a limited number of views or a restricted scale of spaces. In this paper, we introduce Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes. Annotations are provided with surface reconstructions, camera poses, and 2D and 3D semantic segmentations. The precise global alignment and comprehensive, diverse panoramic set of views over entire buildings enable a variety of supervised and self-supervised computer vision tasks, including keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification.


Format your response as a table containing question ID, score, and reasoning." #REFR Under this schema, we obtained the following results: The Vicuna benchmark shows consistent improvement over the base OPT models.

"You are a sophisticated and critical judge evaluating the outputs of AI models.

For each question given above, evaluate the "answer" as a response to the prompt.

Judge each answer on a score of 1-10 based on relevance, coherence, and on its informational value as a response to the question.

Actions should only be considered useful if their output is also present in the answer.

Generate both this score and your reasoning for the score.

The 1.3b OPT model seems to be better, on the surface than the 2.7b model. We don't know why this is.

2.7b was more curt, more prone to insulting the user, refusing to elaborate on answers.

It's interesting that the Eluwa 1.3 2-epoch model is doing better here than base OPT 6.7b (a model 5 times its size).

However, a word of caution: 1.3's verbosity seems to come from regurgitation, and less refined ability to write, and indeed in our testing we found this family of models answering questions with fabricated blogposts, right down to authors' bios and twitter credentials.

Better Question-Answering Models On A Budget

Abstract: Low-rank adaptation (LoRA) and question-answer datasets from large language models have made it much easier for much smaller models to be finetuned to the point where they display sophisticated conversational abilities. In this paper, we present Eluwa, a family of LoRA models that use the Stanford Alpaca dataset and massively improve the capabilities of Facebook's OPT 1.3B, 2.7B and 6.7B models. We benchmark these models in multiple ways, including letting GPT-4 judge their answers to prompts that span general knowledge, writing, programming and other tasks. We show that smaller models here can be fine-tuned to be as performant as models 3x larger-all for as little as 40 USD in compute.

8-Bit Approximations For Parallelism In Deep Learning

Abstract: The creation of practical deep learning data-products often requires parallelization across processors and computers to make deep learning feasible on large data sets, but bottlenecks in communication bandwidth make it difficult to attain good speedups through parallelism. Here we develop and test 8-bit approximation algorithms which make better use of the available bandwidth by compressing 32-bit gradients and nonlinear activations to 8-bit approximations. We show that these approximations do not decrease predictive performance on MNIST, CIFAR10, and ImageNet for both model and data parallelism and provide a data transfer speedup of 2x relative to 32-bit parallelism. We build a predictive model for speedups based on our experimental data, verify its validity on known speedup data, and show that we can obtain a speedup of 50x and more on a system of 96 GPUs compared to a speedup of 23x for 32-bit. We compare our data types with other methods and show that 8-bit approximations achieve state-of-the-art speedups for model parallelism. Thus 8-bit approximation is an efficient method to parallelize convolutional networks on very large systems of GPUs.


Following the work #REFR , we split all labeled frames into subject-exclusive 3-fold with 12 AUs for both two stages.

The proposed CLEF is compared with the state-of-theart methods on six popular databases for FER and AUR tasks.

Furthermore, we conduct ablation studies to verify the component-wise contribution of our method.

Expert coders select the most expressive 20s of each video clip for AU coding, producing 140,000 labeled frames.

BP4D+ #OTHEREFR consists of 140 subjects with total of 1.5 M frames in the same laboratory environments.

For each subject, 20 seconds from 4 activities are annotated, resulting in 192,000 labeled frames.

First, the 140 subjects are split into four fold, following the same setting in #OTHEREFR .

In pretraining, we equally sample 480,000 frames from all 1.5 M frames by 10 activity categories.

In fine-tuning, 12 AUs, the same as in BP4D, are selected for AU recognition.

Weakly-Supervised Text-Driven Contrastive Learning For Facial Behavior Understanding

Abstract: Contrastive learning has shown promising potential for learning robust representations by utilizing unlabeled data. However, constructing effective positive-negative pairs for contrastive learning on facial behavior datasets remains challenging. This is because such pairs inevitably encode the subject-ID information, and the randomly constructed pairs may push similar facial images away due to the limited number of subjects in facial behavior datasets. To address this issue, we propose to utilize activity descriptions, coarse-grained information provided in some datasets, which can provide high-level semantic information about the image sequences but is often neglected in previous studies. More specifically, we introduce a two-stage Contrastive Learning with Text-Embeded framework for Facial behavior understanding (CLEF). The first stage is a weakly-supervised contrastive learning method that learns representations from positive-negative pairs constructed using coarse-grained activity information. The second stage aims to train the recognition of facial expressions or facial action units by maximizing the similarity between image and the corresponding text label names. The proposed CLEF achieves state-of-the-art performance on three inthe-lab datasets for AU recognition and three in-the-wild datasets for facial expression recognition. Recently, many deep learning based approaches [28, 53, 38, 41, 55, 18, 4] have been proposed and achieved state-ofthe-art performance in FER and AUR. A variety of meth

Eac-Net: A Region-Based Deep Enhancing And Cropping Approach For Facial Action Unit Detection

Abstract: In this paper, we propose a deep learning basedapproach for facial action unit detection by enhancing andcropping the regions of interest. The approach is implementedby adding two novel nets (layers): the enhancing layers and thecropping layers, to a pretrained CNN model. For the enhancinglayers (the E-Net), we designed an attention map based on faciallandmark features and applied it to a pretrained neural networkto conduct enhanced learning. For the cropping layers (the CNet), we crop facial regions around the detected landmarksand design convolutional layers to learn deeper features foreach facial region. We then fuse the E-Net and the C-Net toobtain our Enhancing and Cropping (EAC) Net, which canlearn both feature enhancing and region cropping functions.Our approach shows significant improvement in performancecompared to the state-of-the-art methods applied to BP4D andDISFA AU datasets.


The first few eigenfunctions corresponding to small eigenvalues (the convergence depends on the decay of the spectral problems #REFR ) are considered as the multiscale basis functions.

These approaches are usually referred to as multiscale methods in the literature, among which we may mention: Multiscale Finite Element Method #OTHEREFR , the Variational Multiscale Method #OTHEREFR , Mixed Multiscale Finite Element Method #OTHEREFR , Mixed Mortar Multiscale Finite Element Method #OTHEREFR , the two-scale Finite Element Method #OTHEREFR , and the Multiscale Finite Volume method #OTHEREFR .

The aforementioned methods share model reduction techniques, using different structures to find multiscale solutions, especially in many practical applications such as fluid flow simulations, for instance, #OTHEREFR .

In particular, we consider a family of an extended version of MsFEM, known as the Generalized Multiscale Finite Element Method (GMsFEM) that was first introduced by #OTHEREFR .

The main idea of GMsFEM is to construct localized basis functions by solving local spectral problems that are used to approximate the solution on a coarse grid incorporating fine-scale features.

Following this, we construct an auxiliary space associated with local spectral problems in the coarse grid.

In this paper, we extend the Constraint Energy Minimization Generalized Multiscale Finite Element Method (CEM-GMsFEM) developed in #OTHEREFR to solve single-phase nonlinear compressible flow.

The key ideas of the method can be summarized as follows: First, we construct the auxiliary basis functions by solving local spectral problems. Then, by using oversampling techniques and localization (cf.

#OTHEREFR ), we solve an appropriate energy subject to some constrainted oversampling regions to find the required basis functions.

Finally, the resulting basis functions are shown to have exponential decay away from the target coarse element, and therefore, they are localizable.

Then we rigorously analyze the convergence error estimates for the proposed scheme.

Convergence Of The Cem-Gmsfem For Compressible Flow In Highly Heterogeneous Media

Abstract: This paper presents and analyses a Constraint Energy Minimization Generalized Multiscale Finite Element Method (CEM-GMsFEM) for solving single-phase nonlinear compressible flows in highly heterogeneous media. The construction of CEM-GMsFEM hinges on two crucial steps: First, the auxiliary space is constructed by solving local spectral problems, where the basis functions corresponding to small eigenvalues are captured. Then the basis functions are obtained by solving local energy minimization problems over the oversampling domains using the auxiliary space. The basis functions have exponential decay outside the corresponding local oversampling regions. The convergence of the proposed method is provided, and we show that this convergence only depends on the coarse grid size and is independent of the heterogeneities. An online enrichment guided by a posteriori error estimator is developed to enhance computational efficiency. Several numerical experiments on a three-dimensional case to confirm the theoretical findings are presented, illustrating the performance of the method and giving efficient and accurate numerical

Generalized Multiscale Finite Element Methods (Gmsfem)

Abstract: In this paper, we propose a general approach called Generalized Multiscale Finite Element Method (GMsFEM) for performing multiscale simulations for problems without scale separation over a complex input space. As in multiscale finite element methods (MsFEMs), the main idea of the proposed approach is to construct a small dimensional local solution space that can be used to generate an efficient and accurate approximation to the multiscale solution with a potentially high dimensional input parameter space. In the proposed approach, we present a general procedure to construct the offline space that is used for a systematic enrichment of the coarse solution space in the online stage. The enrichment in the online stage is performed based on a spectral decomposition of the offline space. In the online stage, for any input parameter, a multiscale space is constructed to solve the global problem on a coarse grid. The online space is constructed via a spectral decomposition of the offline space and by choosing the eigenvectors corresponding to the largest eigenvalues. The computational saving is due to the fact that the construction of the online multiscale space for any input parameter is fast and this space can be re-used for solving the forward problem with any forcing and boundary condition. Compared with the other approaches where global snapshots are used, the local approach that we present in this paper allows us to eliminate unnecessary degrees of freedom on a coarse-grid level. We present various examples in the paper and some numerical results to demonstrate the effectiveness of our method.


Replacing k by N b in the inequality #REFR leads to the inequality in the proposition.

The inequality #OTHEREFR has been obtained similar to #OTHEREFR .

Let (ŷ, y) with the maximum value of L be convex and γ-Lipschitz.

Assume Adam is run for T iterations with a learning rate η and batch size b to obtain f B S ,R .

Then we have the following upper bound for E(f B S ,R ) with probability at least 1 − δ:

Lipschitzness Effect Of A Loss Function On Generalization Performance Of Deep Neural Networks Trained By Adam And Adamw Optimizers

Abstract: The generalization performance of deep neural networks with regard to the optimization algorithm is one of the major concerns in machine learning. This performance can be affected by various factors. In this paper, we theoretically prove that the Lipschitz constant of a loss function is an important factor to diminish the generalization error of the output model obtained by Adam or AdamW. The results can be used as a guideline for choosing the loss function when the optimization algorithm is Adam or AdamW. In addition, to evaluate the theoretical bound in a practical setting, we choose the human age estimation problem in computer vision. For assessing the generalization better, the training and test datasets are drawn from different distributions. Our experimental evaluation shows that the loss function with lower Lipschitz constant and maximum value improves the generalization of the model trained by Adam or AdamW.

A Pac-Bayesian Approach To Spectrally-Normalized Margin Bounds For Neural Networks

Abstract: We present a generalization bound for feedforward neural networks in terms of the product of the spectral norm of the layers and the Frobenius norm of the weights. The generalization bound is derived using a PAC-Bayes analysis.


Split learning #REFR emerges as a potential solution: without sharing its raw data, the edge feeds the private inputs through the first few layers of DNNs and transmits the intermediate features to the cloud to offload partial computation.

Recent years have witnessed remarkable growth in deep learning applications, as deep neural networks (DNNs) have grown deeper and larger.

It poses a dilemma for the thin edge device: on one hand, it lacks the computational power to individually train the models; on the other, it would violate data privacy sending all data to an untrusted party, e.g., the cloud, to process.

However, the vanilla split learning still faces privacy leakage since an adversary could infer the input from the feature #OTHEREFR .

Hence many works have proposed to hide sensitive information from the features, such as encryption #OTHEREFR , adversarial learning #OTHEREFR , differential privacy [6] , etc.

However, these works mostly sacrifice either accuracy or efficiency for privacy: crypto-based solutions are often far inefficient; privacy-preserving methods in the plaintext are efficient, but always face the tradeoffs between accuracy and privacy.

Our work presents the first privacy-preserving split learning paradigm without any accuracy loss in plaintext.

We discovered many DNN modules have a natural structure that is permutation equivalent, meaning that they can process permuted inputs equivalently to the original inputs.

Shuffled Transformer For Privacy-Preserving Split Learning

Abstract: In conventional split learning, training and testing data often face severe privacy leakage threats. Existing solutions often have to trade learning accuracy for data privacy, or the other way around. We propose a lossless privacy-preserving split learning framework, on the basis of the permutation equivalence properties which are inherent to many neural network modules. We adopt Transformer as the example building block to the framework. It is proved that the Transformer encoder block is permutation equivalent, and thus training/testing could be done equivalently on permuted data. We further introduce shufflingbased privacy guarantee and enhance it by mix-up training. All properties are verified by conducted experiments, which also show strong defence against privacy attacks compared to the stateof-the-art, yet without any accuracy decline.

Distributed Learning Of Deep Neural Network Over Multiple Agents

Abstract: In domains such as health care and finance, shortage of labeled data and computational resources is a critical issue while developing machine learning algorithms. To address the issue of labeled data scarcity in training and deployment of neural network-based systems, we propose a new technique to train deep neural networks over several data sources. Our method allows for deep neural networks to be trained using data from multiple entities in a distributed fashion. We evaluate our algorithm on existing datasets and show that it obtains performance which is similar to a regular neural network trained on a single machine. We further extend it to incorporate semi-supervised learning when training with few labeled samples, and analyze any security concerns that may arise. Our algorithm paves the way for distributed training of deep neural networks in data sensitive applications when raw data may not be shared directly.


We initialize the text encoder from a relatively small XLM-R Base #REFR model with 270M parameters.

Table 4 shows a comparison with state-of-the-art methods on low-shot IN1k and iNat18, including foundational and self-supervised models.

Our models show impressive lowshot performance on both IN1k and iNat18, reaching 83.7% and 80.3% top-1 accuracy with only 10 labeled examples per class, and reaching the highest reported performance with just one labeled example per class on IN1k of 62.1%.

Strong foundational models are expected to also have a good open world understanding of visual concepts.

To equip our pretrained vision encoders with such capabilities, we utilize LiT #OTHEREFR .

Table 5 shows the zero-shot transfer performance of our models on IN1k, and F-101.

Our ViT-2B attains 80.8% accuracy on IN1k, outperforming a similarly sized OpenCLIP model.

Our results lag behind other works which pretrain on datasets such as JFT-3B and ALIGN, highlighting the importance of the pretraining Table 6 : Detection and Segmentation results on LVIS (v1 val).

We compare MAE→WSP with prior work and report the detection (AP box ) and instance segmentation performance (AP mask ).

Our ViT-H significantly outperforms SWAG which uses the same pretraining and model size, but without pre-pretraining.

The Effectiveness Of Mae Pre-Pretraining For Billion-Scale Pretraining

Abstract: This paper revisits the standard pretrain-then-finetune paradigm used in computer vision for visual recognition tasks. Typically, state-of-the-art foundation models are pretrained using large scale (weakly) supervised datasets with billions of images. We introduce an additional prepretraining stage that is simple and uses the self-supervised MAE technique to initialize the model. While MAE has only been shown to scale with the size of models, we find that it scales with the size of the training dataset as well. Thus, our MAE-based pre-pretraining scales with both model and data size making it applicable for training foundation models. Pre-pretraining consistently improves both the model convergence and the downstream transfer performance across a range of model scales (millions to billions of parameters), and dataset sizes (millions to billions of images). We measure the effectiveness of pre-pretraining on 10 different visual recognition tasks spanning image classification, video recognition, object detection, low-shot classification and zero-shot recognition. Our largest model achieves new state-of-theart results on iNaturalist-18 (91.3%), 1-shot ImageNet-1k (62.1%), and zero-shot transfer on Food-101 (96.0%). Our study reveals that model initialization plays a significant role, even for web-scale pretraining with billions of images. * Equal technical contribution. † Project Lead.

Unsupervised Cross-Lingual Representation Learning At Scale

Abstract: This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +13.8% average accuracy on XNLI, +12.3% average F1 score on MLQA, and +2.1% average F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 11.8% in XNLI accuracy for Swahili and 9.2% for Urdu over the previous XLM model. We also present a detailed empirical evaluation of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-Ris very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make XLM-R code, data, and models publicly available.


Although this representation can explain the prediction somewhat, it can still be abstract to users. To solve this problem, Park et al. #REFR and Wu et al.

Since these applications are closely related to the safety and property of our life, there are high requirements for the reliability of the VQA methods.

Although most present methods have demonstrated exemplary performance, they are considered black boxes, as their internal logic and workings are concealed from users, which cannot meet the reliability requirements.

It is considered that providing a reasonable explanation for the prediction process can become one of the solutions to prove the model's reliability.

Therefore, new regulations have been created to establish standards for verifying mandatory decisions, which increases the demand for methods' interpretability #OTHEREFR .

Most conventional interpretable VQA models use a heat map to represent the different attention parts of the model in an image.

#OTHEREFR have proposed interpretable VQA models to generate explanations with human-readable and detailed natural language sentences.

These methods require creating another model that generates explanations for the VQA model.

Due to the generation of explanations and answers being independent, it is difficult to control whether the generated explanation conforms to the logic used by the VQA model.

Recently, pre-trained large-scale models such as the Bidirectional Encoder Representations from Transformers (BERT) #OTHEREFR , GPT-2 #OTHEREFR , and the Contrastive Language-Image Pre-training (CLIP) #OTHEREFR have been developed rapidly, which makes it possible to generate natural language more correctly.

Focusing on the deficiency of the previous models, Sammani et al. #OTHEREFR , Marasovic et al. #OTHEREFR , and Kayser et al.

Interpretable Visual Question Answering Referring To Outside Knowledge

Abstract: We present a novel multimodal interpretable VQA model that can answer the question more accurately and generate diverse explanations. Although researchers have proposed several methods that can generate human-readable and fine-grained natural language sentences to explain a model's decision, these methods have focused solely on the information in the image. Ideally, the model should refer to various information inside and outside the image to correctly generate explanations, just as we use background knowledge daily. The proposed method incorporates information from outside knowledge and multiple image captions to increase the diversity of information available to the model. The contribution of this paper is to construct an interpretable visual question answering model using multimodal inputs to improve the rationality of generated results. Experimental results show that our model can outperform state-ofthe-art methods regarding answer accuracy and explanation rationality.

Multimodal Explanations: Justifying Decisions And Pointing To The Evidence

Abstract: Deep models that are both effective and explainable are desirable in many settings; prior explainable models have been unimodal, offering either image-based visualization of attention weights or text-based generation of post-hoc justifications. We propose a multimodal approach to explanation, and argue that the two modalities provide complementary explanatory strengths. We collect two new datasets to define and evaluate this task, and propose a novel model which can provide joint textual rationale generation and attention visualization. Our datasets define visual and textual justifications of a classification decision for activity recognition tasks (ACT-X) and for visual question answering tasks (VQA-X). We quantitatively show that training with the textual explanations not only yields better textual justification models, but also better localizes the evidence that supports the decision. We also qualitatively show cases where visual explanation is more insightful than textual explanation, and vice versa, supporting our thesis that multimodal explanation models offer significant benefits over unimodal approaches.


MGCN #REFR , though not showing distinct advantages on other datasets, shows a competitive result as those pre-training techniques on QM7 and QM8 datasets.

The overall model performance on the MoleculeNet dataset also indicates that self-supervised pre-training tasks result in a performance boost.

While supervised pre-training tasks introduce the risk of negative transfer #OTHEREFR , well-designed self-supervised pre-trained models show rival performance in learning implicit domain knowledge #OTHEREFR - #OTHEREFR .

However, some tasks that involve alteration of the molecule structure, such as subgraph removal #OTHEREFR , result in poor performance in datasets sensitive to topology changes, such as the BBBP dataset.

Moreover, the composition of different augmentation strategies is not bound to perform better since removing various substructures might easily lead to the loss of important structural information.

Furthermore, models involving 3D information have superior performance on datasets of quantum mechanical properties.

For retrosynthesis prediction, RetroXpert #OTHEREFR and Reaction-Aware #OTHEREFR combine the information from both graph representation and sequence representation and perform well. Generally speaking, template-free models have better performance. However, LocalRetro #OTHEREFR , a template-based model, outperforms others.

This may be due to the model design, which not only strictly constrains the templates but also considers global structure.

For reaction prediction, models such as GTPN #OTHEREFR , Molecular Transformer #OTHEREFR and NERF #OTHEREFR which utilize the state-of-art models in other applications of machine learning also get good results in this task.

Deep Learning Methods For Small Molecule Drug Discovery: A Survey

Abstract: With the development of computer-assisted techniques, research communities including biochemistry and deep learning have been devoted into the drug discovery field for over a decade. Various applications of deep learning have drawn great attention in drug discovery, such as molecule generation, molecular property prediction, retrosynthesis prediction, and reaction prediction. While most existing surveys only focus on one of the applications, limiting the view of researchers in the community. In this paper, we present a comprehensive review on the aforementioned four aspects, and discuss the relationships among different applications. The latest literature and classical benchmarks are presented for better understanding the development of variety of approaches. We commence by summarizing the molecule representation format in these works, followed by an introduction of recent proposed approaches for each of the four tasks. Furthermore, we review a variety of commonly used datasets and evaluation metrics and compare the performance of deep learning-based models. Finally, we conclude by identifying remaining challenges and discussing the future trend for deep learning methods in drug discovery. Impact Statement-As artificial intelligence continues to evolve, an increasing number of deep learning algorithms are proposed to utilize extensive experimental data to accelerate the timeconsuming and costly procedure for drug discovery. Unlike previous surveys that tend to focus on a single aspect of drug discovery, we offer a more comprehensive survey that encompasses four areas of drug discovery utilizing deep learning techniques, including molecule generation, molecular property prediction, retrosynthesis, and reaction prediction. Before introducing the main methods, we start with molecular data representation, which is also crucial for deep learning algorithms. Moreover, we present the recent benchmarks for deep learning-based drug discovery, which cover 9 widely used databases. Index Terms-deep learning, drug discovery.

Molecular Property Prediction: A Multilevel Quantum Interactions Modeling Perspective

Abstract: Predicting molecular properties (e.g., atomization energy) is an essential issue in quantum chemistry, which could speed up much research progress, such as drug designing and substance discovery. Traditional studies based on density functional theory (DFT) in physics are proved to be time-consuming for predicting large number of molecules. Recently, the machine learning methods, which consider much rule-based information, have also shown potentials for this issue. However, the complex inherent quantum interactions of molecules are still largely underexplored by existing solutions. In this paper, we propose a generalizable and transferable Multilevel Graph Convolutional neural Network (MGCN) for molecular property prediction. Specifically, we represent each molecule as a graph to preserve its internal structure. Moreover, the well-designed hierarchical graph neural network directly extracts features from the conformation and spatial information followed by the multilevel interactions. As a consequence, the multilevel overall representations can be utilized to make the prediction. Extensive experiments on both datasets of equilibrium and off-equilibrium molecules demonstrate the effectiveness of our model. Furthermore, the detailed results also prove that MGCN is generalizable and transferable for the prediction.


In contrast to other analysis tools, this leads to fewer false alarms, e.g., when encountering manual reentrancy locking #REFR .

For example, the vulnerability depicted in Figure 6 is detected neither by Confuzzius #OTHEREFR nor by Smartian #OTHEREFR .

However, since EF CF simulates multiple attacker-controlled accounts, it will quickly generate a transaction originating from the first account and leaking the Ether to a second attacker-controlled address.

Most analysis tools feature explicit Reentrancy bug oracles.

In contrast, EF CF does not feature an explicit detector for reentrancy but simply generates reentrant transaction sequences that trigger other bug oracles, such as Ether gains.

In this way, EF CF detects reentrancy bugs, but only if they are actually exploitable.

By default, EF CF reports unprotected selfdestruct only if the selfdestruct will transfer the remaining Ether of the target contract to the attacker, i.e., the address parameter of the selfdestruct is controlled by the attacker.

Optionally, EF CF can also report DoS-style unprotected selfdestructs, i.e., if the selfdestruct can be triggered by anyone, but always targets a trusted address such as the owner.

This style of detection is featured in most other analysis tools.

With Solidity versions > 0.8, contracts feature automatic integer overflow checking and proper assertion violation reporting.

EF CF supports this new Solidity exception mechanism to signal errors to the fuzzer.

Ef Cf: High Performance Smart Contract Fuzzing For Exploit Generation

Abstract: Smart contracts are increasingly being used to manage large numbers of high-value cryptocurrency accounts. There is a strong demand for automated, efficient, and comprehensive methods to detect security vulnerabilities in a given contract. While the literature features a plethora of analysis methods for smart contracts, the existing proposals do not address the increasing complexity of contracts. Existing analysis tools suffer from false alarms and missed bugs in today's smart contracts that are increasingly defined by complexity and interdependencies. To scale accurate analysis to modern smart contracts, we introduce EF CF, a high-performance fuzzer for Ethereum smart contracts. In contrast to previous work, EF CF efficiently and accurately models complex smart contract interactions, such as reentrancy and cross-contract interactions, at a very high fuzzing throughput rate. To achieve this, EF CF transpiles smart contract bytecode into native C++ code, thereby enabling the reuse of existing, optimized fuzzing toolchains. Furthermore, EF CF increases fuzzing efficiency by employing a structure-aware mutation engine for smart contract transaction sequences and using a contract's ABI to generate valid transaction inputs. In a comprehensive evaluation, we show that EF CF scales better-without compromising accuracy-to complex contracts compared to state-of-theart approaches, including other fuzzers, symbolic/concolic execution, and hybrid approaches. Moreover, we show that EF CF can automatically generate transaction sequences that exploit reentrancy bugs to steal Ether.

Sereum: Protecting Existing Smart Contracts Against Re-Entrancy Attacks

Abstract: Recently, a number of existing blockchain systems have witnessed major bugs and vulnerabilities within smart contracts. Although the literature features a number of proposals for securing smart contracts, these proposals mostly focus on proving the correctness or absence of a certain type of vulnerability within a contract, but cannot protect deployed (legacy) contracts from being exploited. In this paper, we address this problem in the context of re-entrancy exploits and propose a novel smart contract security technology, dubbed Sereum (Secure Ethereum), which protects existing, deployed contracts against re-entrancy attacks in a backwards compatible way based on run-time monitoring and validation. Sereum does neither require any modification nor any semantic knowledge of existing contracts. By means of implementation and evaluation using the Ethereum blockchain, we show that Sereum covers the actual execution flow of a smart contract to accurately detect and prevent attacks with a false positive rate as small as 0.06% and with negligible run-time overhead. As a by-product, we develop three advanced re-entrancy attacks to demonstrate the limitations of existing offline vulnerability analysis tools.


Experiment 3: Open-World Detection Setting

The OWOD problem has recently gained popularity with evaluations being performed on the challenging MS COCO dataset #REFR .

Lastly, we also validate the generalizability of our feature enrichment and the STO module for open-world object detection (OWOD) in images.

To this end, we integrate our feature enrichment and STO module (replacing 3D convolutions with 2D) into the recent transformersbased OWOD framework, named OW-DETR #OTHEREFR . Tab.

3 shows the results on the challenging MS COCO split introduced in #OTHEREFR .

For a fair comparison, we employ the same self-supervised ResNet50 backbone as in the standard OW-DETR #OTHEREFR .

We report the results of ORE-EBUI #OTHEREFR and our baseline OW-DETR from #OTHEREFR .

In the case of Task-1, our approach achieves an impressive performance particularly on the 'Unknown' object class by increasing the U-Recall from 5.7% to 8.8%, while also improving the detection performance on the 'Current Known' classes.

Video Instance Segmentation In An Open-World

Abstract: Existing video instance segmentation (VIS) approaches generally follow a closed-world assumption, where only seen category instances are identified and spatio-temporally segmented at inference. Open-world formulation relaxes the close-world static-learning assumption as follows: (a) first, it distinguishes a set of known categories as well as labels an unknown object as 'unknown' and then (b) it incrementally learns the class of an unknown as and when the corresponding semantic labels become available. We propose the first open-world VIS approach, named OW-VISFormer, that introduces a novel feature enrichment mechanism and a spatio-temporal objectness (STO) module. The feature enrichment mechanism based on a lightweight auxiliary network aims at accurate pixel-level (unknown) object delineation from the background as well as distinguishing category-specific known semantic classes. The STO module strives to generate instance-level pseudolabels by enhancing the foreground activations through a contrastive loss. Moreover, we also introduce an extensive experimental protocol to measure the characteristics of OW-VIS. Our OW-VISFormer performs favorably against a solid baseline in OW-VIS setting. Further, we evaluate our contributions in the standard fully-supervised VIS setting by integrating them into the recent SeqFormer, achieving an absolute gain of 1.6% AP on Youtube-VIS 2019 val. set. Lastly, we show the generalizability of our contributions for the open-world detection (OWOD) setting, outperforming the best existing OWOD method in the literature.

Microsoft Coco: Common Objects In Context

Abstract: We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.


They fine-tuned mBERT model #REFR in a crosslingual setup and in a multi-task setup, and found that the results are mixed in terms of their benefits.

There exist several studies on multilingual and/or multi-task learning in the context of media analysis. #OTHEREFR

(2021) applied cross-lingual training to multilingual false information detection and showed that cross-lingual training results in on par performance to monolingual training. Alam et al.

(2021a,b) annotated Tweets in multiple languages with multiple choice questions regarding COVID-19 disinformation.

The SemEval 2023 Task 3 dataset (Piskorski et al., 2023) exhibits significantly different properties to the datasets previous works used and we found that cross-lingual and/or multi-task learning help in this dataset.

We also carried out more thorough experiments than the previous works, which might have been the key to the improvement.

There also exist different approaches for multilingual and low-resource settings in broader domains.

#OTHEREFR proposed a method for training multilingual sentence embeddings and demonstrated its effectiveness in 50+ languages. In a low-resource setup, Heinisch et al.

(2022) used a data augmentation approach that retrieves additional data from related datasets by automatically labeling them.

Hitachi At Semeval-2023 Task 3: Exploring Cross-Lingual Multi-Task Strategies For Genre And Framing Detection In Online News

Abstract: This paper explains the participation of team Hitachi to SemEval-2023 Task 3 "Detecting the genre, the framing, and the persuasion techniques in online news in a multilingual setup." Based on the multilingual, multi-task nature of the task and the low-resource setting, we investigated different cross-lingual and multi-task strategies for training the pretrained language models. Through extensive experiments, we found that (a) cross-lingual/multi-task training, and (b) collecting an external balanced dataset, can benefit the genre and framing detection. We constructed ensemble models from the results and achieved the highest macro-averaged F1 scores in Italian and Russian genre categorization subtasks. The target articles are those identified to be

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


Approach Using A Modified Wta Assumption

With the new WTA assumption, the learning analysis for general hierarchies follows the same pattern as the analysis for tree hierarchies in #REFR , and yields the same time bound.

Therefore, they satisfy the WTA requirement to be candidate neurons, provided that they have strictly more than o • k incoming neighbors that fire.

But we know that, during the learning process, they have k incoming neighbors that fire (in the noise-free case), which is > o • k by the assumption that o < 1.

A similar argument holds for the noisy case, based on having at least pk incoming neighbors that fire, which is > o • k by the assumption that o < p.

For neurons that were not previously engaged, Lemma 7.4, Property 1, implies that the incoming potential is exactly kw.

For the single neuron that was previously engaged for c, Lemma 7.4, Property 2 implies that the incoming potential is strictly greater than kw. So the WTA rule selects the previously-engaged neuron.

Implementing Assumption 7.3 will require some additional mechanism, in addition to the mechanisms that are used to implement the basic WTA satisfying Assumption 7.2.

Such a mechanism could serve as a pre-processing step, before the basic WTA.

The new mechanism could allow a layer neuron u to fire (and somehow reflect its incoming potential) exactly if u has strictly more than o • k incoming neighbors that contribute potential ≥ w to u. 13

Alternative modifications to the WTA: The modification in Section 7.2.2 allows the WTA to explicitly reject candidate neurons having too few incoming neighbors whose edge weight is greater than or equal to the initial weight w. There might be other ways to modify the WTA.

For example, we might assume that the WTA can remember when it has already selected a neuron to represent a particular concept, and not select it again for a different concept.

Learning Hierarchically-Structured Concepts Ii: Overlapping Concepts, And Networks With Feedback

Abstract: We continue our study from [5], of how concepts that have hierarchical structure might be represented in brain-like neural networks, how these representations might be used to recognize the concepts, and how these representations might be learned. In [5], we considered simple tree-structured concepts and feed-forward layered networks. Here we extend the model in two ways: we allow limited overlap between children of different concepts, and we allow networks to include feedback edges. For these more general cases, we describe and analyze algorithms for recognition and algorithms for learning. We also use the abbreviations supp r,f (B, * , t) for S(, t), supp r,f (B, , *) for t S(, t), and supp r,f (B, * , *) for ,t S(, t), Notice that each of these three unions must converge to a finite set since all the sets S(, t) are subsets of the single finite set C of concepts. Now we have two monotonicity results, for r and f : Lemma 2.5. Let C be any concept hierarchy satisfying the limited-overlap property, and let B ⊆ D 0. Consider r, r , where 0 ≤ r ≤ r ≤ 1, and arbitrary f. Then supp r ,f (B) ⊆ supp r,f (B). Lemma 2.6. Let C be any concept hierarchy satisfying the limited-overlap property, and let B ⊆ D 0. Consider f, f , where 0 ≤ f ≤ f , and arbitrary r ∈ [0, 1]. Then supp r,f (B) ⊆ supp r,f (B).

Learning Hierarchically Structured Concepts

Abstract: We study the question of how concepts that have structure get represented in the brain. Specifically, we introduce a model for hierarchically structured concepts and we show how a biologically plausible neural network can recognize these concepts, and how it can learn them in the first place. Our main goal is to introduce a general framework for these tasks and prove formally how both (recognition and learning) can be achieved. ::: We show that both tasks can be accomplished even in presence of noise. For learning, we analyze Oja's rule formally, a well-known biologically-plausible rule for adjusting the weights of synapses. We complement the learning results with lower bounds asserting that, in order to recognize concepts of a certain hierarchical depth, neural networks must have a corresponding number of layers.


The convergence rates are of the same order as the convergence rates of the deterministic forward-backward algorithm over R d #REFR .

α W 2 2 (µ N ,π) ≤ ε 2 , and F(p 2N ) − F(π) ≤ ε 2 .

Proof. The proof is given in Appendix E.3.

Theorem 5.2 states the sublinear convergence of FB-GVI for a convex V (in terms of objective gap) and Theorem 5.4 states the linear convergence of FB-GVI for a strongly convex V .

Finally, we also extend our results to the non-convex case, where we obtain a stationary point guarantee.

Theorem 5.5 (Non-convex case, FB-GVI). Suppose that V is β-smooth, and

that 0 < η ≤ 1 β . Let ∆ := F(p 0 ) − F(π).

Then, min k∈{0,...,N −1} ∇ BW F(p k ) 2 p k ≤ 150∆ ηN .

In particular, when η = 1 β and N β∆ ε 2 , we obtain the guarantee

Forward-Backward Gaussian Variational Inference Via Jko In The Bures-Wasserstein Space

Abstract: Variational inference (VI) seeks to approximate a target distribution π by an element of a tractable family of distributions. Of key interest in statistics and machine learning is Gaussian VI, which approximates π by minimizing the Kullback-Leibler (KL) divergence to π over the space of Gaussians. In this work, we develop the (Stochastic) Forward-Backward Gaussian Variational Inference (FB-GVI) algorithm to solve Gaussian VI. Our approach exploits the composite structure of the KL divergence, which can be written as the sum of a smooth term (the potential) and a non-smooth term (the entropy) over the Bures-Wasserstein (BW) space of Gaussians endowed with the Wasserstein distance. For our proposed algorithm, we obtain state-of-the-art convergence guarantees when π is log-smooth and log-concave, as well as the first convergence guarantees to first-order stationary solutions when π is only log-smooth.

A Unified Theory Of Sgd: Variance Reduction, Sampling, Quantization And Coordinate Descent

Abstract: In this paper we introduce a unified analysis of a large family of variants of proximal stochastic gradient descent ({\tt SGD}) which so far have required different intuitions, convergence analyses, have different applications, and which have been developed separately in various communities. We show that our framework includes methods with and without the following tricks, and their combinations: variance reduction, importance sampling, mini-batch sampling, quantization, and coordinate sub-sampling. As a by-product, we obtain the first unified theory of {\tt SGD} and randomized coordinate descent ({\tt RCD}) methods, the first unified theory of variance reduced and non-variance-reduced {\tt SGD} methods, and the first unified theory of quantized and non-quantized methods. A key to our approach is a parametric assumption on the iterates and stochastic gradients. In a single theorem we establish a linear convergence result under this assumption and strong-quasi convexity of the loss function. Whenever we recover an existing method as a special case, our theorem gives the best known complexity result. Our approach can be used to motivate the development of new useful methods, and offers pre-proved convergence guarantees. To illustrate the strength of our approach, we develop five new variants of {\tt SGD}, and through numerical experiments demonstrate some of their properties.


Table 10 shows the metrics of the fitted mesh on the SURREAL #REFR test set.

This may be because the arm has larger variations in the dataset.

In order to disentangle the ability of mesh regression from markers usingÂ with the ability to detect the virtual markers accurately from images, we first compute the estimation errors of the virtual markers.

The MPJPE over all the virtual markers is 35.5mm, which demonstrates that these virtual markers can be accurately detected from the images.

We then fit the mesh model parameters to these virtual markers.

As we can see, the fitted mesh has a similar error as our regression ones which uses the interpolation matrixÂ, which validates the accuracy of the estimated virtual markers.

Figure 12 shows more qualitative comparisons with Pose2Mesh #OTHEREFR on the SURREAL test set in which has diverse body shapes.

The skeleton representation used in Pose2Mesh loses the body shape information so the method #OTHEREFR can only recover mean shapes.

For example, in Figure 12 (d) (e), the estimated meshes of Pose2Mesh tend to have the average body shape and fail to estimate the real body shape, regardless of whether the person is slim or stout.

This is caused by the limited skeleton representation bottleneck so that the model learns a mean shape for the whole training dataset implicitly.

3D Human Mesh Estimation From Virtual Markers

Abstract: Inspired by the success of volumetric 3D pose estimation, some recent human mesh estimators propose to estimate 3D skeletons as intermediate representations, from which, the dense 3D meshes are regressed by exploiting the mesh topology. However, body shape information is lost in extracting skeletons, leading to mediocre performance. The advanced motion capture systems solve the problem by placing dense physical markers on the body surface, which allows to extract realistic meshes from their non-rigid motions. However, they cannot be applied to wild images without markers. In this work, we present an intermediate representation, named virtual markers, which learns 64 landmark keypoints on the body surface based on the large-scale mocap data in a generative style, mimicking the effects of physical markers. The virtual markers can be accurately detected from wild images and can reconstruct the intact meshes with realistic shapes by simple interpolation. Our approach outperforms the state-of-the-art methods on three datasets. In particular, it surpasses the existing methods by a notable margin on the SURREAL dataset, which has diverse body shapes. Code is available at https: //github.com/ShirleyMaxx/VirtualMarker.

Learning From Synthetic Humans

Abstract: Estimating human pose, shape, and motion from images and videos are fundamental challenges with many applications. Recent advances in 2D human pose estimation use large amounts of manually-labeled training data for learning convolutional neural networks (CNNs). Such data is time consuming to acquire and difficult to extend. Moreover, manual labeling of 3D pose, depth and motion is impractical. In this work we present SURREAL (Synthetic hUmans foR REAL tasks): a new large-scale dataset with synthetically-generated but realistic images of people rendered from 3D sequences of human motion capture data. We generate more than 6 million frames together with ground truth pose, depth maps, and segmentation masks. We show that CNNs trained on our synthetic dataset allow for accurate human depth estimation and human part segmentation in real RGB images. Our results and the new dataset open up new possibilities for advancing person analysis using cheap and large-scale synthetic data.


In a recent paper of Guruswami, Nakkiran, and Sudan #REFR , they consider hidden Markov channels, where the states evolve according to some Markov process.

The problem of designing coding schemes for stochastic channels is hard.

The "hidden" refers to the fact that one cannot determine the state from the output of the channel.

One important thing to note here is that error introduced by the channel is only dependent on the state of the Markov process.

In designing encoding and decoding algorithms for the problem of hidden Markov channels, Markovian sources become quite important due to a well-known correspondence relating error-correction for additive Markov chains to compression and decompression algorithms for Markovian sources (when the compression is linear).

Recall that additive channels are the those which map inputs from some alphabet Σ to outputs over Σ, where there is an abelian group defined on Σ (needed for the additive purposes).

The channel creates an error sequence from the associated Markov source which is independent of the input sequence.

Improving The Runtime Of Algorithmic Polarization Of Hidden Markov Models

Abstract: We improve the runtime of the linear compression scheme for hidden Markov sources presented in a 2018 paper of Guruswami, Nakkiran, and Sudan. Under the previous scheme, compressing a message of length n takes O(n log n) runtime, and decompressing takes O(n 1+δ) runtime for any fixed δ > 0. We present how to improve the runtime of the decoding scheme to O(n log n) by caching intermediate results to avoid repeating computation.

Algorithmic Polarization For Hidden Markov Models

Abstract: Using a mild variant of polar codes we design linear compression schemes compressing Hidden Markov sources (where the source is a Markov chain, but whose state is not necessarily observable from its output), and to decode from Hidden Markov channels (where the channel has a state and the error introduced depends on the state). We give the first polynomial time algorithms that manage to compress and decompress (or encode and decode) at input lengths that are polynomial $\it{both}$ in the gap to capacity and the mixing time of the Markov chain. Prior work achieved capacity only asymptotically in the limit of large lengths, and polynomial bounds were not available with respect to either the gap to capacity or mixing time. Our results operate in the setting where the source (or the channel) is $\it{known}$. If the source is $\it{unknown}$ then compression at such short lengths would lead to effective algorithms for learning parity with noise -- thus our results are the first to suggest a separation between the complexity of the problem when the source is known versus when it is unknown.


The work of #REFR introduced two metrics called "localization tightness" and "localization stability" to measure the informativeness of object localization.

(b) Combine uncertainty-based scores and category re-weighting approach to evaluate the informative region score (Sec. 4.3).

(c) Then, a scale-aware selection method is used to select the diverse regions and allocate labeling budgets for each scale (Sec. 4.4).

(d) The selected regions are annotated from a human expert, and subsequently updated to the labeled set (Sec. 4.5). The process repeats for a specified number of cycles.

fication scores as metrics to measure model uncertainty, as seen in works such as #OTHEREFR .

#OTHEREFR proposed an entropybased Non-Maximum Suppression method that considers both instance-level uncertainty and diversity.

Furthermore, some researchers have focused on combining the uncertainty of localization with the uncertainty of classification.

For example, [4] proposed a mixture density network to aggregate these two sources of uncertainty, and #OTHEREFR considered the consistency of localization and predicted class distributions in augmented and unlabeled images.

Fine-grained active learning strategies query partial labels within specific instances or regions, instead of selecting the entire image.

For instance, #OTHEREFR proposed a self-supervised sample mining technique that uses pseudolabeling with high consistency samples and an active learning strategy to select low consistency samples within crossimage validation.

[6] employed the coreset method #OTHEREFR and classification uncertainty to select instance labels.

Mural: Multi-Scale Region-Based Active Learning For Object Detection

Abstract: Obtaining large-scale labeled object detection dataset can be costly and time-consuming, as it involves annotating images with bounding boxes and class labels. Thus, some specialized active learning methods have been proposed to reduce the cost by selecting either coarse-grained samples or fine-grained instances from unlabeled data for labeling. However, the former approaches suffer from redundant labeling, while the latter methods generally lead to training instability and sampling bias. To address these challenges, we propose a novel approach called Multiscale Region-based Active Learning (MuRAL) for object detection. MuRAL identifies informative regions of various scales to reduce annotation costs for well-learned objects and improve training performance. The informative region score is designed to consider both the predicted confidence of instances and the distribution of each object category, enabling our method to focus more on difficult-to-detect classes. Moreover, MuRAL employs a scale-aware selection strategy that ensures diverse regions are selected from different scales for labeling and downstream finetuning, which enhances training stability. Our proposed method surpasses all existing coarse-grained and fine-grained baselines on Cityscapes and MS COCO datasets, and demonstrates significant improvement in difficult category performance.

Localization-Aware Active Learning For Object Detection

Abstract: Active learning - a class of algorithms that iteratively searches for the most informative samples to include in a training dataset - has been shown to be effective at annotating data for image classification. However, the use of active learning for object detection is still largely unexplored as determining informativeness of an object-location hypothesis is more difficult. In this paper, we address this issue and present two metrics for measuring the informativeness of an object hypothesis, which allow us to leverage active learning to reduce the amount of annotated data needed to achieve a target object detection performance. Our first metric measures 'localization tightness' of an object hypothesis, which is based on the overlapping ratio between the region proposal and the final prediction. Our second metric measures 'localization stability' of an object hypothesis, which is based on the variation of predicted object locations when input images are corrupted by noise. Our experimental results show that by augmenting a conventional active-learning algorithm designed for classification with the proposed metrics, the amount of labeled training data required can be reduced up to 25%. Moreover, on PASCAL 2007 and 2012 datasets our localization-stability method has an average relative improvement of 96.5% and 81.9% over the baseline method using classification only.


We remark that the notion of robustness studied in #REFR was quite different from that of the previous works as detailed below.

The two topics, of stable matching and the design of algorithms that produce solutions that are robust to errors, have been studied extensively for decades and there are today several books on each of them, e.g., see #OTHEREFR and #OTHEREFR .

Yet, there is a paucity of results at the intersection of these two topics.

Indeed, before the publication of #OTHEREFR , we are aware of only a few previous works [ABG + 16, ABF + 17, CSS21].

[ABG + 16] considered the problem of finding stable matching under uncertain linear preferences. They proposed three different uncertainty models:

Lottery Model: Each agent has a probability distribution over strict preference lists, independent of other agents.

A Structural And Algorithmic Study Of Stable Matching Lattices Of Multiple Instances

Abstract: Recently [MV18a] identified and initiated work on the new problem of understanding structural relationships between the lattices of solutions of two "nearby" instances of stable matching. They also gave an application of their work to finding a robust stable matching. However, the types of changes they allowed in going from instance A to B were very restricted, namely any one agent executes an upward shift. In this paper, we allow any one agent to permute its preference list arbitrarily. Let M A and M B be the sets of stable matchings of the resulting pair of instances A and B, and let L A and L B be the corresponding lattices of stable matchings. We prove that the matchings in M A ∩ M B form a sublattice of both L A and L B and those in M A \ M B form a join semi-sublattice of L A. These properties enable us to obtain a polynomial time algorithm for not only finding a stable matching in M A ∩ M B , but also for obtaining the partial order, as promised by Birkhoff's Representation Theorem [Bir37], thereby enabling us to generate all matchings in this sublattice. Our algorithm also helps solve a version of the robust stable matching problem. We discuss another potential application, namely obtaining new insights into the incentive compatibility properties of the Gale-Shapley Deferred Acceptance Algorithm.

Finding Stable Matchings That Are Robust To Errors In The Input

Abstract: We study the problem of finding solutions to the stable matching problem that are robust to errors in the input and we obtain a polynomial time algorithm for a special class of errors. In the process, we also initiate work on a new structural question concerning the stable matching problem, namely finding relationships between the lattices of solutions of two"nearby"instances. Our main algorithmic result is the following: We identify a polynomially large class of errors, $D$, that can be introduced in a stable matching instance. Given an instance $A$ of stable matching, let $B$ be the random variable that represents the instance that results after introducing {\em one} error from $D$, chosen via a given discrete probability distribution. The problem is to find a stable matching for $A$ that maximizes the probability of being stable for $B$ as well. Via new structural properties of the type described in the question stated above, we give a combinatorial polynomial time algorithm for this problem. We also show that the set of robust stable matchings for instance $A$, under probability distribution $p$, forms a sublattice of the lattice of stable matchings for $A$. We give an efficient algorithm for finding a succinct representation for this set; this representation has the property that any member of the set can be efficiently retrieved from it.


FML #REFR proposes a self-supervised method to learn identity and appearance models from videos.

Recently, deep learning approaches use the 3DMM fitting as prior and learn additional corrective displacements or normal maps #OTHEREFR , in order to capture missing facial details.

DECA #OTHEREFR and EMOCA #OTHEREFR first predict the parameters of FLAME #OTHEREFR , and then learn a UV-map of vertex displacements. Tran et al.

#OTHEREFR borrow details from reference bump maps in order to complete occluded facial regions.

Several subject-specific methods leverage multi-view images #OTHEREFR or learn implicit representations using multi-layer perceptrons (MLPs) #OTHEREFR . However, these methods cannot generalize to multiple identities.

Monocular videobased optimization techniques [24-27, 57, 60] leverage the multi-frame consistency to learn facial details.

However, all these videoonly methods ignore the audio modality that is usually included in most cases.

In addition, they do not explicitly handle occlusions that frequently appear in talking face videos.

Another line of work addresses the problem of audio-driven 3D facial animation. Busso et al.

#OTHEREFR use hidden Markov models (HMMs) to model the temporal relation between prosodic audio features and head motion sequences.

Avface: Towards Detailed Audio-Visual 4D Face Reconstruction

Abstract: Figure 1. Given a monocular talking face video and the corresponding speech segment, AVFace reconstructs the detailed 4D face geometry for any input speaker, and it is robust in cases when either modality is insufficient.

Fml: Face Model Learning From Videos

Abstract: Monocular image-based 3D reconstruction of faces is a long-standing problem in computer vision. Since image data is a 2D projection of a 3D face, the resulting depth ambiguity makes the problem ill-posed. Most existing methods rely on data-driven priors that are built from limited 3D face scans. In contrast, we propose multi-frame video-based self-supervised training of a deep network that (i) learns a face identity model both in shape and appearance while (ii) jointly learning to reconstruct 3D faces. Our face model is learned using only corpora of in-the-wild video clips collected from the Internet. This virtually endless source of training data enables learning of a highly general 3D face model. In order to achieve this, we propose a novel multi-frame consistency loss that ensures consistent shape and appearance across multiple frames of a subject's face, thus minimizing depth ambiguity. At test time we can use an arbitrary number of frames, so that we can perform both monocular as well as multi-frame reconstruction.


Due to the equivalence of active user identification and GT, we use an adaptation of the GT decoding strategy in #REFR to the context of user identification in non-coherent ( , k)-MnAC.

Proof: Using standard capacity arguments #OTHEREFR along with Lemma 1, it is evident that user identification cost in the non-coherent ( , k)-MnAC must obey n( )

where C is as given in Lemma 1. Thus, in the k = Θ(1) regime we have

Next, we present a decoder as shown in Algorithm 1 which can achieve the equality in #OTHEREFR relying on classical thresholding techniques in information theory #OTHEREFR .

The algorithm operates as follows: Given the set of preambles P and the detector outputs Z, Number of threshold tests passed, n t = 0.

Non-Coherent Active Device Identification For Massive Random Access

Abstract: Massive Machine-Type Communications (mMTC) is a key service category in the current generation of wireless networks featuring an extremely high density of energy and resource-limited devices with sparse and sporadic activity patterns. In order to enable random access in such mMTC networks, base station needs to identify the active devices while operating within stringent access delay constraints. In this paper, an energy efficient active device identification protocol is proposed in which active devices transmit On-Off Keying (OOK) modulated preambles jointly and base station employs non-coherent energy detection avoiding channel estimation overheads. The minimum number of channel-uses required by the active user identification protocol is characterized in the asymptotic regime of total number of devices when the number of active devices k scales as k = Θ(1) along with an achievability scheme relying on the equivalence of activity detection to a group testing problem. Several practical schemes based on Belief Propagation (BP) and Combinatorial Orthogonal Matching Pursuit (COMP) are also proposed. Simulation results show that BP strategies outperform COMP significantly and can operate close to the theoretical achievability bounds. In a partial-recovery setting where few misdetections are allowed, BP continues to perform well.

Boolean Compressed Sensing And Noisy Group Testing

Abstract: The fundamental task of group testing is to recover a small distinguished subset of items from a large population while efficiently reducing the total number of tests (measurements). The key contribution of this paper is in adopting a new information-theoretic perspective on group testing problems. We formulate the group testing problem as a channel coding/decoding problem and derive a single-letter characterization for the total number of tests used to identify the defective set. Although the focus of this paper is primarily on group testing, our main result is generally applicable to other compressive sensing models.

active user identification
user identification


We warm up the training for 4 epochs and then we decay the learning rate using a cosine schedule #REFR .

We apply the basic data augmentations (only random cropping, flipping and data whitening) to the input images when computing the student features.

We empirically find out more advanced data augmentations (such as "rand-m9-n3-mstd0.5") harm the distillation performance.

We argue that the advanced data augmentations for the student model's input images enlarge the discrepancy of the student feature and the fixed pre-extracted teacher feature, which lead to the poor performance in vision-language knowledge distillation. We train all models for 100 epochs.

During the training, we use the Adam optimizer #OTHEREFR with decoupled weight decay regularization [57] .

We set the initial learning rate as 8 × 10 −4 and weight decay 0.05.

We remove the weight decay of the weights that are the gains or biases.

For our model Distill-ViT-B/32, we optimize with the batch size 12288 for images and texts. For Distill-UniCL*, we adopt 8192 as the batch size.

We grid search for the best hyperparameters µ vl , µ p-vl and µ udist when performing ablation studies on losses in Sec 4.5 (main paper).

We finally set µ vl = 100, µ p-vl = 33.3 and µ udist = 14.3 for all experiments.

Dime-Fm : Distilling Multimodal And Efficient Foundation Models

Abstract: Large Vision-Language Foundation Models (VLFM), such as CLIP, ALIGN and Florence, are trained on largescale datasets of image-caption pairs and achieve superior transferability and robustness on downstream tasks, but they are difficult to use in many practical applications due to their large size, high latency and fixed architectures. Unfortunately, recent work shows training a small custom VLFM for resource-limited applications is currently very difficult using public and smaller-scale data. In this paper, we introduce a new distillation mechanism (DIME-FM) that allows us to transfer the knowledge contained in large VLFMs to smaller, customized foundation models using a relatively small amount of inexpensive, unpaired images and sentences. We transfer the knowledge from the pre-trained CLIP-ViT-L/14 model to a ViT-B/32 model, with only 40M public images and 28.4M unpaired public sentences. The resulting model "Distill-ViT-B/32" rivals the CLIP-ViT-B/32 model pre-trained on its private WiT dataset (400M image-text pairs): Distill-ViT-B/32 achieves similar results in terms of zero-shot and linear-probing performance on both Ima-geNet and the ELEVATER (20 image classification tasks) benchmarks. It also displays comparable robustness when evaluated on five datasets with natural distribution shifts from ImageNet.

Sgdr: Stochastic Gradient Descent With Warm Restarts

Abstract: Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at https://github.com/loshchil/SGDR


The face images for this dataset are originally from Flickr-Faces-HQ (FFHQ) dataset #REFR which provides a large human face images.

Also, the fact that there are many large face datasets available that can be beneficial for further studies.

besides, face is an important human attribute as progresses in AI

resulted in advanced synthetic human face image generator models.

This phenomenon makes it more important to have countermeasures to avoid malicious activities in MFs #OTHEREFR .

Some sample images of FFHQ dataset are shown in Figure 1 .

They provided images given from multiple printers as reported in the Table 1 .

All printers are laser type, and 200 images are used from each.

They printed 200 images on fifty A4 papers, each paper including 4 images, using landscape orientation.

They assumed that 200 images may not be enough to deploy them for many deep learning techniques, so they provided another set of images which included Regions of Interest (ROI) extracted from the original images.

D R A F T Spritz-Ps: Validation Of Synthetic Face Images Using A Large Dataset Of Printed Documents

Abstract: The capability of doing effective forensic analysis on printed and scanned (PS) images is essential in many applications. PS documents may be used to conceal the artifacts of images which is due to the synthetic nature of images since these artifacts are typically present in manipulated images and the main artifacts in the synthetic images can be removed after the PS. Due to the appeal of Generative Adversarial Networks (GANs), synthetic face images generated with GANs models are difficult to differentiate from genuine human faces and may be used to create counterfeit identities. Additionally, since GANs models do not account for physiological constraints for generating human faces and their impact on human IRISes, distinguishing genuine from synthetic IRISes in the PS scenario becomes extremely difficult. As a result of the lack of large-scale reference IRIS datasets in the PS scenario, we aim at developing a novel dataset to become a standard for Multimedia Forensics (MFs) investigation which is available at [45]. In this paper, we provide a novel dataset made up of a large number of synthetic and natural printed IRISes taken from VIPPrint Printed and Scanned face images. We extracted irises from face images and it is possible that the model due to eyelid occlusion captured the incomplete irises. To fill the missing pixels of extracted iris, we applied techniques to discover the complex link between the iris images. To highlight the problems involved with the evaluation of the dataset's IRIS images, we conducted a large number of analyses employing Siamese Neural Networks to assess the similarities between genuine and synthetic human IRISes, such as ResNet50, Xception, VGG16, and MobileNet-v2. For instance, using the Xception network, we achieved 56.76% similarity of IRISes for synthetic images and 92.77% similarity of IRISes for real images.

A Style-Based Generator Architecture For Generative Adversarial Networks

Abstract: We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.


Consistent with prior work #REFR the skill discovery problem is formalized as a Markov Decision Process (MDP) M = (S, A, P) without external rewards, S and A respectively denote the state and action spaces, and P is the transition function. Skills introduced by Sutton et al.

#OTHEREFR are temporally extended actions (sub-behavior), which consist of a sequence of primitive actions.

We define skills as policies ( | , ) conditioned on a fixed latent variable ∈ .

Skill discovery methods aim to learn these latent-conditioned policies by maximising the mutual information between S and .

Due to symmetry, the corresponding mutual information can be expressed in two forms:

(S; ) = ( ) − ( |S) reverse = (S) − (S| ) forward (1)

Controlled Diversity With Preference : Towards Learning A Diverse Set Of Desired Skills

Abstract: Autonomously learning diverse behaviors without an extrinsic reward signal has been a problem of interest in reinforcement learning. However, the nature of learning in such mechanisms is unconstrained, often resulting in the accumulation of several unusable, unsafe or misaligned skills. In order to avoid such issues and ensure the discovery of safe and human-aligned skills, it is necessary to incorporate humans into the unsupervised training process, which remains a largely unexplored research area. In this work, we propose Controlled Diversity with Preference (CDP) 1 , a novel, collaborative human-guided mechanism for an agent to learn a set of skills that is diverse as well as desirable. The key principle is to restrict the discovery of skills to those regions that are deemed to be desirable as per a preference model trained using human preference labels on trajectory pairs. We evaluate our approach on 2D navigation and Mujoco environments and demonstrate the ability to discover diverse, yet desirable skills.

Explore, Discover And Learn: Unsupervised Discovery Of State-Covering Skills

Abstract: Acquiring abilities in the absence of a task-oriented reward function is at the frontier of reinforcement learning research. This problem has been studied through the lens of empowerment, which draws a connection between option discovery and information theory. Information-theoretic skill discovery methods have garnered much interest from the community, but little research has been conducted in understanding their limitations. Through theoretical analysis and empirical evidence, we show that existing algorithms suffer from a common limitation -- they discover options that provide a poor coverage of the state space. In light of this, we propose 'Explore, Discover and Learn' (EDL), an alternative approach to information-theoretic skill discovery. Crucially, EDL optimizes the same information-theoretic objective derived from the empowerment literature, but addresses the optimization problem using different machinery. We perform an extensive evaluation of skill discovery methods on controlled environments and show that EDL offers significant advantages, such as overcoming the coverage problem, reducing the dependence of learned skills on the initial state, and allowing the user to define a prior over which behaviors should be learned. Code is publicly available at https://github.com/victorcampos7/edl.


In the brain science field, it has been shown that brain dynamics has a core-periphery organization #REFR .

Core-periphery structure represents a relationship between nodes in a graph where the core nodes are densely connected with each other while periphery nodes are sparsely connected to the core nodes and among each other #OTHEREFR , #OTHEREFR .

Core-periphery graph has been applied in a variety of fields, including social network analysis #OTHEREFR , #OTHEREFR , economics #OTHEREFR , biology such as modeling the structure of protein interaction networks #OTHEREFR .

The functional brain networks also demonstrate a coreperiphery structure #OTHEREFR .

A recent study revealed the coreperiphery characteristics of the human brain from a structural perspective #OTHEREFR .

It is shown that gyri and sulci, two prominent cortical folding patterns, could cooperate as a core-periphery network which improves the efficiency of information transmission in the brain #OTHEREFR .

Cp-Cnn: Core-Periphery Principle Guided Convolutional Neural Network

Abstract: The evolution of convolutional neural networks (CNNs) can be largely attributed to the design of its architecture, i.e., the network wiring pattern. Neural architecture search (NAS) advances this by automating the search for the optimal network architecture, but the resulting network instance may not generalize well in different tasks. To overcome this, exploring network design principles that are generalizable across tasks is a more practical solution. In this study, We explore a novel brain-inspired design principle based on the core-periphery property of the human brain network to guide the design of CNNs. Our work draws inspiration from recent studies suggesting that artificial and biological neural networks may have common principles in optimizing network architecture. We implement the core-periphery principle in the design of network wiring patterns and the sparsification of the convolution operation. The resulting core-periphery principle guided CNNs (CP-CNNs) are evaluated on three different datasets. The experiments demonstrate the effectiveness and superiority compared to CNNs and ViT-based methods. Overall, our work contributes to the growing field of brain-inspired AI by incorporating insights from the human brain into the design of neural networks.

Task-Based Core-Periphery Organization Of Human Brain Dynamics

Abstract: As a person learns a new skill, distinct synapses, brain regions, and circuits are engaged and change over time. In this paper, we develop methods to examine patterns of correlated activity across a large set of brain regions. Our goal is to identify properties that enable robust learning of a motor skill. We measure brain activity during motor sequencing and characterize network properties based on coherent activity between brain regions. Using recently developed algorithms to detect time-evolving communities, we find that the complex reconfiguration patterns of the brain's putative functional modules that control learning can be described parsimoniously by the combined presence of a relatively stiff temporal core that is composed primarily of sensorimotor and visual regions whose connectivity changes little in time and a flexible temporal periphery that is composed primarily of multimodal association regions whose connectivity changes frequently. The separation between temporal core and periphery changes over the course of training and, importantly, is a good predictor of individual differences in learning success. The core of dynamically stiff regions exhibits dense connectivity, which is consistent with notions of core-periphery organization established previously in social networks. Our results demonstrate that core-periphery organization provides an insightful way to understand how putative functional modules are linked. This, in turn, enables the prediction of fundamental human capacities, including the production of complex goal-directed behavior.


Regularization Effect On Robustness And Performance

The proof when uncertainty set is (16) is similar to proof of Theorem 1 in #REFR .

We show the proof when the uncertainty set is (15).

max ∆Φ∈U y − (Φ + ∆Φ)β 2 ≤ y − Φβ 2 + max ∆Φ∈U ∆Φβ 2 ≤ y − Φβ 2 + max ∆Φ∈U ∆Φ 2 β 2 = y − Φβ 2 + c β 2 2 Now, let u = y − Φβ y − Φβ 2 , and let ∆Φ * i −cβ i sgn (β i ) u, we have ∆Φ * 2 = c β 2 , therefore ∆Φ * ∈ U.

Meanwhile, max ∆Φ∈U y − (Φ + ∆Φ)β 2 ≥ max ∆Φ∈U y − (Φ + ∆Φ * )β 2 = y − Φβ + c β 2 2 u 2 = y − Φβ 2 + c β 2 2 .

This completes the proof of the theorem.

Here we do not make a definitive conclusion on which regularization method is better.

The theoretical results are based on different assumptions that are usually hard to verify in practice.

Enhancing Robustness Of Gradient-Boosted Decision Trees Through One-Hot Encoding And Regularization *

Abstract: Gradient-boosted decision trees (GBDT) are widely used and highly effective machine learning approach for tabular data modeling. However, their complex structure may lead to low robustness against small covariate perturbation in unseen data. In this study, we apply one-hot encoding to convert a GBDT model into a linear framework, through encoding of each tree leaf to one dummy variable. This allows for the use of linear regression techniques, plus a novel risk decomposition for assessing the robustness of a GBDT model against covariate perturbations. We propose to enhance the robustness of GBDT models by refitting their linear regression forms with L 1 or L 2 regularization. Theoretical results are obtained about the effect of regularization on the model performance and robustness. It is demonstrated through numerical experiments that the proposed regularization approach can enhance the robustness of the one-hot-encoded GBDT models. * The views expressed in the paper are those of the authors and do not represent the views of Wells Fargo.

Robust Regression And Lasso

Abstract: Lasso, or l1 regularized least squares, has been explored extensively for its remarkable sparsity properties. In this paper it is shown that the solution to Lasso, in addition to its sparsity, has robustness properties: it is the solution to a robust optimization problem. This has two important consequences. First, robustness provides a connection of the regularizer to a physical property, namely, protection from noise. This allows a principled selection of the regularizer, and in particular, generalizations of Lasso that also yield convex optimization problems are obtained by considering different uncertainty sets. Second, robustness can itself be used as an avenue for exploring different properties of the solution. In particular, it is shown that robustness of the solution explains why the solution is sparse. The analysis as well as the specific results obtained differ from standard sparsity results, providing different geometric intuition. Furthermore, it is shown that the robust optimization formulation is related to kernel density estimation, and based on this approach, a proof that Lasso is consistent is given, using robustness directly. Finally, a theorem is proved which states that sparsity and algorithmic stability contradict each other, and hence Lasso is not stable.