Word vector would be obtained from this component by converting text into low-dimensional dense vectors based on pre-trained Bert model #REFR .

For skill extraction from job description, the model train on the AutoDL platform [27] and it will be loaded on my laptop.

As it is considered as classification task in Bert, token [CLS] would be added to the start of the sentence and token [SEP] would be added to the end of the sentence.

Generated input id is obtained by looking up words in the pre-trained Bert model and replacing it with sub-word and specific token, which is based on WordPiece algorithm.

The processed input sentence, generated input id, will be fed into Bert model.

A list of skills id related to the job description will be provided and remain the first 20 th skills id, which will be changed into skills label.

Table Of Contents

Abstract: Due to the increasing number of graduates, many applicants experience the situation about finding a job, and employers experience difficulty filtering job applicants, which might negatively impact their effectiveness. However, most job-hunting websites lack job recommendation and CV filtering or ranking functionality, which are not integrated into the system. Thus, an smart job hunter combined with the above functionality will be conducted in this project, which contains job recommendations, CV ranking and even a job dashboard for skills and job applicant's functionality. Job recommendation and CV ranking starts from the automatic keyword extraction and end with the Job/CV ranking algorithm. Automatic keyword extraction is implemented by Job2Skill and the CV2Skill model based on Bert. Job2Skill consists of two components, text encoder and Gru-based layers, while CV2Skill is mainly based on Bert and fine-tunes the pre-trained model by the Resume-Entity dataset. Besides, to match skills from CV and job description and rank lists of job and candidates, job/CV ranking algorithms have been provided to compute the occurrence ratio of skill words based on TFIDF score and match ratio of the total skill numbers. Besides, some advanced features have been integrated into the website to improve user experiences, such as the calendar and sweetalert2 plugin. And some basic features to go through job application processes, such as job application tracking and interview arrangement. This paper has been divided into seven parts. The first part introduces the aims, objectives and challenges of the project. The second part provided this project's three requirements: essential, recommended and optional. The third part introduces the background of the project and some related techniques that will be used in the project. The fourth part describes the project's system design, including the frontend, backend, model architecture and ranking algorithm. The fifth part shows the details of implementing the functionality and automatic keyword extraction. The sixth part describes the evaluation methods for the website, and the last part summarises the project and future work.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


As the original DeiT is trained on strong augmentation, such as RandAugment or 3-Augment, we also compare our method with "weak augmented" ViT-S, where it only employs resized random crop (RRC) and CutMix #REFR .

We compare ViT-S models trained on ImageNet-1k with different training strategies using robustness benchmarks.

We employ three scenarios: (1) noise and blur scenario (2) domain shift scenario (3) adversarial attack scenario.

For the first scenario, we add Gaussian noise and Gaussian blur to the validation images.

We use ImageNet-R #OTHEREFR and Sketch-ImageNet #OTHEREFR for testing the robustness against domain shifts.

Finally, we use a weak version of AutoAttak #OTHEREFR for measuring adversarial robustness.

Our assumption is that because the pixel-trained models are sensitive to imperceptible details, they will be less robust than our approach in noise or adversarial attack scenarios.

However, on the other hand, because our method relies on the encoding power of the pre-trained tokenizer, if the employed tokenizer is not a robust feature extractor, our method could be more vulnerable than pixel-trained counterparts.

Table B .4 shows the results of the first and the second scenarios. Here, we observe two important findings.

First, when we use the same augmentations with the same strength (ViT-S Weak Aug vs.

ViT-S SeiT), SeiT shows smaller performance drops on both noise scenarios and do- main shift scenarios.

Seit: Storage-Efficient Vision Training With Tokens Using 1% Of Pixel Storage

Abstract: We need billion-scale images to achieve more generalizable and groundbreaking vision models, as well as massive dataset storage to ship the images (e.g., the LAION-4B dataset needs 240TB storage space). However, it has become challenging to deal with unlimited dataset storage with limited storage infrastructure. A number of storageefficient training methods have been proposed to tackle the problem, but they are rarely scalable or suffer from severe damage to performance. In this paper, we propose a storage-efficient training strategy for vision classifiers for large-scale datasets (e.g., ImageNet) that only uses 1024 tokens per instance without using the raw level pixels; our token storage only needs <1% of the original JPEG-compressed raw pixels. We also propose token augmentations and a Stem-adaptor module to make our approach able to use the same architecture as pixel-based approaches with only minimal modifications on the stem layer and the carefully tuned optimization settings. Our experimental results on ImageNet-1k show that our method significantly outperforms other storage-efficient training methods with a large gap. We further show the effectiveness of our method in other practical scenarios, storage-efficient pre-training, and continual learning. Code is available at https://github.com/naver-ai/seit Adjusting image resolution Adjusting JPEG quality C-score based sampling Uniform random sampling Original DeiT SeiT SeiT (pixel fine-tuned) Storage (GB, log scale) Top 1 Accuracy (%)

Cutmix: Regularization Strategy To Train Strong Classifiers With Localizable Features

Abstract: Regional dropout strategies have been proposed to enhance the performance of convolutional neural network classifiers. They have proved to be effective for guiding the model to attend on less discriminative parts of objects (e.g. leg as opposed to head of a person), thereby letting the network generalize better and have better object localization capabilities. On the other hand, current methods for regional dropout remove informative pixels on training images by overlaying a patch of either black pixels or random noise. Such removal is not desirable because it leads to information loss and inefficiency during training. We therefore propose the CutMix augmentation strategy: patches are cut and pasted among training images where the ground truth labels are also mixed proportionally to the area of the patches. By making efficient use of training pixels and retaining the regularization effect of regional dropout, CutMix consistently outperforms the state-of-the-art augmentation strategies on CIFAR and ImageNet classification tasks, as well as on the ImageNet weakly-supervised localization task. Moreover, unlike previous augmentation methods, our CutMix-trained ImageNet classifier, when used as a pretrained model, results in consistent performance gains in Pascal detection and MS-COCO image captioning benchmarks. We also show that CutMix improves the model robustness against input corruptions and its out-of-distribution detection performances. Source code and pretrained models are available at this https URL .


Clinical Natural Language Processing Beyond

To mitigate this English-centric problem, some datasets only sample questions from native speakers of each langauge #REFR .

Though there are variants that use crosslingual supervision (e.g., #OTHEREFR , many of these polyglot models can benefit from joint training of different languages without any explicit supervision.

We suspect that similar polyglot language modeling is happenning in LLMs, such as ChatGPT and GPT-4, which we tested on our IGAKU QA benchmark in Japanese, a language typologically distant from English.

Open-Domain and Multilingual Question Answering Much prior work proposed datasets for open-domain QA for English and beyond #OTHEREFR .

Several works pointed out the problem of translation-based question answering evaluations #OTHEREFR : questions raised mainly by English speakers can diverge from information needs from speakers of other languages.

For instance, these translationbased benchmarks can overly represent Englishcentric topics, such as American politics, sports, and culture.

Consistent with such data creation methods, our IGAKU QA consists of problems that are written by native Japanese speakers to evaluate the qualifica-tions and skills for medical practice in the country.

Evaluating Gpt-4 And Chatgpt On Japanese Medical Licensing Examinations

Abstract: As large language models (LLMs) gain popularity among speakers of diverse languages, we believe that it is crucial to benchmark them to better understand model behaviors, failures, and limitations in languages beyond English. In this work, we evaluate LLM APIs (Chat-GPT, GPT-3, and GPT-4) on the Japanese national medical licensing examinations from the past five years, including the current year. Our team comprises native Japanese-speaking NLP researchers and a practicing cardiologist based in Japan. Our experiments show that GPT-4 outperforms ChatGPT and GPT-3 and passes all six years of the exams, highlighting LLMs' potential in a language that is typologically distant from English. However, our evaluation also exposes critical limitations of the current LLM APIs. First, LLMs sometimes select prohibited choices (禁忌肢) that should be strictly avoided in medical practice in Japan, such as suggesting euthanasia. Further, our analysis shows that the API costs are generally higher and the maximum context size is smaller for Japanese because of the way non-Latin scripts are currently tokenized in the pipeline. We release our benchmark as IGAKU QA as well as all model outputs and exam metadata. We hope that our results and benchmark will spur progress on more diverse applications of LLMs.

Tydi Qa: A Benchmark For Information-Seeking Question Answering In Typologically Diverse Languages

Abstract: Confidently making progress on multilingual modeling requires challenging, trustworthy evaluations. We present TyDi QA---a question answering dataset covering 11 typologically diverse languages with 204K question-answer pairs. The languages of TyDi QA are diverse with regard to their typology---the set of linguistic features each language expresses---such that we expect models performing well on this set to generalize across a large number of the world's languages. We present a quantitative analysis of the data quality and example-level qualitative linguistic analyses of observed language phenomena that would not be found in English-only corpora. To provide a realistic information-seeking task and avoid priming effects, questions are written by people who want to know the answer, but don't know the answer yet, and the data is collected directly in each language without the use of translation.


The Impact Of Mini-Batch Size For Simclr And Barlowtwins

This belief was confirmed by one of the Figure in #REFR that display a 7% accuracy gap on ImageNet between a model trained with a small (256) versus large batch size (8192).

2that SimCLR uses negative examples in the denominator term of its loss.

It has been reasoned that the role of this term is to perform negative sampling #OTHEREFR and thus that is will only be effective when the number of samples i.e. mini-batch size is large. Equivalently, BarlowTwins (recall Eq.

(3)) estimates correlation along the sample dimensions and thus also is expected to benefit from large mini-batch size for more accurate estimation.

In this section, we propose to refute that large mini-batch sizes are required to successfully train JE-SSL with these two methods.

The belief that many methods rely on large mini-batch size has been mentioned in many recent studies, e.g. #OTHEREFR .

However, an important point that people often miss when reading #OTHEREFR is the critical influence of the optimizer and the choice of the learning rate when training with different batch size.

In their appendix, #OTHEREFR show that when using a better optimization technique, the gap in performances between bigger and smaller batch size are significantly reduced.

The impact of the downstream task One caveat of the batch size analysis in #OTHEREFR is that it concerns only the performances on ImageNet.

Since one of the main motivation behind SSL is to learn a model whose representations can generalize to different tasks, we analyse the performances with different batch sizes across several downstream tasks: ImageNet-1K [19] , CIFAR10 #OTHEREFR , CLEVR #OTHEREFR , Eurosat #OTHEREFR , Inaturalist #OTHEREFR and Places #OTHEREFR .

In Figure 2 , we plot the performances with SimCLR trained with several batch sizes.

Towards Democratizing Joint-Embedding Self-Supervised Learning

Abstract: Joint Embedding Self-Supervised Learning (JE-SSL) has seen rapid developments in recent years, due to its promise to effectively leverage large unlabeled data. The development of JE-SSL methods was driven primarily by the search for ever increasing downstream classification accuracies, using huge computational resources, and typically built upon insights and intuitions inherited from a close parent JE-SSL method. This has led unwittingly to numerous preconceived ideas that carried over across methods e.g. that SimCLR requires very large mini batches to yield competitive accuracies; that strong and computationally slow data augmentations are required. In this work, we debunk several such ill-formed a priori ideas in the hope to unleash the full potential of JE-SSL free of unnecessary limitations. In fact, when carefully evaluating performances across different downstream tasks and properly optimizing hyper-parameters of the methods, we most often-if not always-see that these widespread misconceptions do not hold. For example we show that it is possible to train SimCLR to learn useful representations, while using a single image patch as negative example, and simple Gaussian noise as the only data augmentation for the positive pair. Along these lines, in the hope to democratize JE-SSL and to allow researchers to easily make more extensive evaluations of their methods, we introduce an optimized PyTorch library for SSL https: //github.com/facebookresearch/FFCV-SSL.

A Simple Framework For Contrastive Learning Of Visual Representations

Abstract: This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.


From Table 3b , we can observe that: 1) Either ComPro use mask attention or segment embedding significantly outperformed SCT #REFR .

"C-CS", "S-CS", "T-CS", and "Cap" denote content-related control signal, structure-related control signal, tense control signal #OTHEREFR , and generated captions, respectively.

Different colors show the corresponding control region sets and the associations between generated words and regions. responding visual guidance after removing masks.

Instead, structure variables designed to learn global information are barely affected.

To validate the effectiveness of mask attention and segment embedding 2 , we run several ablations with content-related control signals.

2) ComPro with mask attention can achieve the best performance, while the model with only segment embedding or both are not as effective and require more training sources.

We evaluated the performance of Com-Pro on structure control and content control by setting different structure prompt lengths M ∈ {1, 10, 40, 80} and content prompt lengths N ∈ {1, 5, 10, 20}, respectively.

From Table 3c , we can observe that: The performance of ComPro keeps improving when we increase the prompt length.

Then, the metrics reach the best scores and even slightly drop with longer prompt length.

Learning Combinatorial Prompts For Universal Controllable Image Captioning

Abstract: Controllable Image Captioning (CIC)-generating natural language descriptions about images under the guidance of given control signals-is one of the most promising directions towards next-generation captioning systems. Till now, various kinds of control signals for CIC have been proposed, ranging from content-related control to structurerelated control. However, due to the format and target gaps of different control signals, all existing CIC works (or architectures) only focus on one certain control signal, and overlook the human-like combinatorial ability. By "combinatorial", we mean that our humans can easily meet multiple needs (or constraints) simultaneously when generating descriptions. To this end, we propose a novel prompt-based framework for CIC by learning Combinatorial Prompts, dubbed as ComPro. Specifically, we directly utilize a pretrained language model GPT-2 [36] as our language model, which can help to bridge the gap between different signalspecific CIC architectures. Then, we reformulate the CIC as a prompt-guide sentence generation problem, and propose a new lightweight prompt generation network to generate the combinatorial prompts for different kinds of control signals. For different control signals, we further design a new mask attention mechanism to realize the prompt-based CIC. Due to its simplicity, our ComPro can easily be extended to more complex combined control signals by concatenating these prompts. Extensive experiments on two prevalent CIC benchmarks have verified the effectiveness and efficiency of our ComPro on both single and combined control signals.

Show, Control And Tell: A Framework For Generating Controllable And Grounded Captions

Abstract: Current captioning approaches can describe images using black-box architectures whose behavior is hardly controllable and explainable from the exterior. As an image can be described in infinite ways depending on the goal and the context at hand, a higher degree of controllability is needed to apply captioning algorithms in complex scenarios. In this paper, we introduce a novel framework for image captioning which can generate diverse descriptions by allowing both grounding and controllability. Given a control signal in the form of a sequence or set of image regions, we generate the corresponding caption through a recurrent architecture which predicts textual chunks explicitly grounded on regions, following the constraints of the given control. Experiments are conducted on Flickr30k Entities and on COCO Entities, an extended version of COCO in which we add grounding annotations collected in a semi-automatic manner. Results demonstrate that our method achieves state of the art performances on controllable image captioning, in terms of caption quality and diversity. Code and annotations are publicly available at: https://github.com/aimagelab/show-control-and-tell.


Raff #REFR directly compared the reported results of 255 AI papers without using the code from the original papers, and found out that 162 papers were at least 75% consistent with reported results while 93 were not.

#OTHEREFR proposed a hybrid classifier to automatically extract open-access datasets and software from scientific papers. Gunderson et al.

#OTHEREFR investigated 400 AI papers, and found that none contain documentation of published experiments, methods, and data altogether.

Although the work referenced above have highlighted the importance of including codes and data alongside published findings, studies were limited to meta-level indicators.

There are a handful of studies that directly compare reproduced results of AI algorithms with published results. For example, Olorisade et al. #OTHEREFR Figure 2 : Workflow of our study.

papers using the codes and datasets from the original papers, reporting inconsistent results with published findings.

Lack of transparency and reproducibility is particularly critical given the standard for AI papers to evaluate performance of proposed methods against baselines.

Such problems have been found in AI research on machine learning analyses on clinical research #OTHEREFR and deep metric learning #OTHEREFR .

Replication studies that provide side-by-side comparisons between AI papers addressing the same topic are rarely conducted.

A Study On Reproducibility And Replicability Of Table Structure Recognition Methods

Abstract: Concerns about reproducibility in artificial intelligence (AI) have emerged, as researchers have reported unsuccessful attempts to directly reproduce published findings in the field. Replicability, the ability to affirm a finding using the same procedures on new data, has not been well studied. In this paper, we examine both reproducibility and replicability of a corpus of 16 papers on table structure recognition (TSR), an AI task aimed at identifying cell locations of tables in digital documents. We attempt to reproduce published results using codes and datasets provided by the original authors. We then examine replicability using a dataset similar to the original as well as a new dataset, GenTSR, consisting of 386 annotated tables extracted from scientific papers. Out of 16 papers studied, we reproduce results consistent with the original in only four. Two of the four papers are identified as replicable using the similar dataset under certain IoU values. No paper is identified as replicable using the new dataset. We offer observations on the causes of irreproducibility and irreplicability. All code and data are available on Codeocean at https://codeocean.com/capsule/6680116/tree.

A Step Toward Quantifying Independently Reproducible Machine Learning Research

Abstract: What makes a paper independently reproducible? Debates on reproducibility center around intuition or assumptions but lack empirical results. Our field focuses on releasing code, which is important, but is not sufficient for determining reproducibility. We take the first step toward a quantifiable answer by manually attempting to implement 255 papers published from 1984 until 2017, recording features of each paper, and performing statistical analysis of the results. For each paper, we did not look at the authors code, if released, in order to prevent bias toward discrepancies between code and paper.


Dataset We evaluate our model's performance on MLQA #REFR , a highly-parallel multilingual extractive QA dataset.

MLQA has QA samples in 7 languages including English, Arabic, German, Spanish, Hindi, Vietnamese and Simplified Chinese.

This dataset has 12,738 extractive QA instances in English and over 5K in target language.

9,019 instances are 4-way parallel, 2,930 are 3-way parallel and 789 are 2way parallel.

MLQA is constructed from Wikipedia articles which have similar meaning in parallel languages.

Labeling work is done by crowd-sourcing and translation is done by professional translators. In total, there are over 46,000 QA instances.

Bridging The Language Gap: Knowledge Injected Multilingual Question Answering

Abstract: Question Answering (QA) is the task of automatically answering questions posed by humans in natural languages. There are different settings to answer a question, such as abstractive, extractive, boolean, and multiple-choice QA. As a popular topic in natural language processing tasks, extractive question answering task (extractive QA) has gained extensive attention in the past few years. With the continuous evolvement of the world, generalized cross-lingual transfer (G-XLT), where question and answer context are in different languages, poses some unique challenges over cross-lingual transfer (XLT), where question and answer context are in the same language. With the boost of corresponding development of related benchmarks, many works have been done to improve the performance of various language QA tasks. However, only a few works are dedicated to the G-XLT task. In this work, we propose a generalized crosslingual transfer framework to enhance the model's ability to understand different languages. Specifically, we first assemble triples from different languages to form multilingual knowledge. Since the lack of knowledge between different languages greatly limits models' reasoning ability, we further design a knowledge injection strategy via leveraging link prediction techniques to enrich the model storage of multilingual knowledge. In this way, we can profoundly exploit rich semantic knowledge. Experiment results on real-world datasets MLQA demonstrate that the proposed method can improve the performance by a large margin, outperforming the baseline method by 13.18%/12.00% F1/EM on average. Our code and models will be open-sourced.

Mlqa: Evaluating Cross-Lingual Extractive Question Answering

Abstract: Question answering (QA) models have shown rapid progress enabled by the availability of large, high-quality benchmark datasets. Such annotated datasets are difficult and costly to collect, and rarely exist in languages other than English, making training QA systems in other languages challenging. An alternative to building large monolingual training datasets is to develop cross-lingual systems which can transfer to a target language without requiring training data in that language. In order to develop such systems, it is crucial to invest in high quality multilingual evaluation benchmarks to measure progress. We present MLQA, a multi-way aligned extractive QA evaluation benchmark intended to spur research in this area. MLQA contains QA instances in 7 languages, namely English, Arabic, German, Spanish, Hindi, Vietnamese and Simplified Chinese. It consists of over 12K QA instances in English and 5K in each other language, with each QA instance being parallel between 4 languages on average. MLQA is built using a novel alignment context strategy on Wikipedia articles, and serves as a cross-lingual extension to existing extractive QA datasets. We evaluate current state-of-the-art cross-lingual representations on MLQA, and also provide machine-translation-based baselines. In all cases, transfer results are shown to be significantly behind training-language performance.


And it is initialized using weights which is trained on ImageNet #REFR , and the parameters of the first two layers are frozen throughout training.

Our approach is implemented under PyTorch 1.8.0 framework on a Intel(R) Xeon(R) Silver 4210R CPU(2.40GHz) along with a Nvidia Geforce RTX 3090GPU. The backbone is modified ResNet-50 #OTHEREFR .

Because the shallow layers of the network are more generalized and high layers are more related to specific tasks.

GOT-10k #OTHEREFR , COCO #OTHEREFR , ImageNet VID #OTHEREFR , and ImageNet DET #OTHEREFR provided the data for the training set.

All of the images are cropped and scaled to 127×127 and 511×511 according to the ground truth provided in the dataset. We utilize corrective loss to train SiamTHN.

The whole network is trained with 20 epcohs using Stochastic Gradient Descent (SGD) with a momentum of 0.9. Batch size is 28.

Learning rate for the first 5 warm-up epochs varies from 0.001 to 0.005; for the following 15 epochs, it ranges from 0.005 to 0.00005.

Siamthn: Siamese Target Highlight Network For Visual Tracking

Abstract: Siamese network based trackers develop rapidly in the field of visual object tracking in recent years. The majority of siamese network based trackers now in use treat each channel in the feature maps generated by the backbone network equally, making the similarity response map sensitive to background influence and hence challenging to focus on the target region. Additionally, there are no structural links between the classification and regression branches in these trackers, and the two branches are optimized separately during training. Therefore, there is a misalignment between the classification and regression branches, which results in less accurate tracking results. In this paper, a Target Highlight Module is proposed to help the generated similarity response maps to be more focused on the target region. To reduce the misalignment and produce more precise tracking results, we propose a corrective loss to train the model. The two branches of the model are jointly tuned with the use of corrective loss to produce more reliable prediction results. Experiments on 5 challenging benchmark datasets reveal that the method outperforms current models in terms of performance, and runs at 38 fps, proving its effectiveness and efficiency.

Imagenet Large Scale Visual Recognition Challenge

Abstract: The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the five years of the challenge, and propose future directions and improvements.


From Figure 8 , we find that 1 ∈ [0.5, 0.7] usually leads to a good balance and achieves superior performance, which is consistent with the findings in #REFR .

To analyze the influence of KL divergence, the sparsity, and variance regularizers in the multi-objective loss, we conduct experiments to compare the performance by changing their coefficients 1 , 2 , and 3 . The results w.r.t.

the three coefficients on Yelp and Book are presented in Figure 8 .

The results on Electronics and Book are similar so that we omit the one on Electronics. From Figure 8 , we have the following observations:

• Increasing 1 of KL divergence is useful to improve the performance.

Actually, as indicated by #OTHEREFR , a large 1 for KL divergence regulates the independence of latent factors to facilitate disentangled representations; while a small 1 helps the model to fit the user interactions better #OTHEREFR .

• The sparsity regularizer is essential to improve the performance because Recall and NDCG drop significantly when 2 is changed from 0.5 to 0.

Besides, 2 cannot be too large, which will limit the representation capability of and .

• The decreased performance from 3 = 1 −4 to 3 = 0 justifies the usefulness of the variance regularizer, which balances the gradient optimization across multiple environments.

Moreover, it should be noted that 3 is expected to be tuned in a small magnitude [0, 1 −3 ] because the stronger gradient regularization will inevitably disturb the normal optimization of parameters.

Causal Disentangled Recommendation Against User Preference Shifts

Abstract: Recommender systems easily face the issue of user preference shifts. User representations will become outof-date and lead to inappropriate recommendations if user preference has shifted over time. To solve the issue, existing work focuses on learning robust representations or predicting the shifting pattern. There lacks a comprehensive view to discover the underlying reasons for user preference shifts. To understand the preference shift, we abstract a causal graph to describe the generation procedure of user interaction sequences. Assuming user preference is stable within a short period, we abstract the interaction sequence as a set of chronological environments. From the causal graph, we find that the changes of some unobserved factors (e.g., becoming pregnant) cause preference shifts between environments. Besides, the fine-grained user preference over categories sparsely affects the interactions with different items. Inspired by the causal graph, our key considerations to handle preference shifts lie in modeling the interaction generation procedure by: 1) capturing the preference shifts across environments for accurate preference prediction, and 2) disentangling the sparse influence from user preference to interactions for accurate effect estimation of preference. To this end, we propose a Causal Disentangled Recommendation (CDR) framework, which captures preference shifts via a temporal variational autoencoder and learns the sparse influence from multiple environments. Specifically, an encoder is adopted to infer the unobserved factors from user interactions while a decoder is to model the interaction generation process. Besides, we introduce two learnable matrices to disentangle the sparse influence from user preference to interactions. Lastly, we devise a multi-objective loss to optimize CDR. Extensive experiments on three datasets show the superiority of CDR in enhancing the generalization ability under user preference shifts.

Variational Autoencoders For Collaborative Filtering

Abstract: We extend variational autoencoders (VAEs) to collaborative filtering for implicit feedback. This non-linear probabilistic model enables us to go beyond the limited modeling capacity of linear factor models which still largely dominate collaborative filtering research. We introduce a generative model with multinomial likelihood and use Bayesian inference to learn this powerful generative model. Despite widespread use in language modeling and economics, the multinomial likelihood receives less attention in the recommender systems literature. We introduce a different regularization parameter for the learning objective, which proves to be crucial for achieving competitive performance. Remarkably, there is an efficient way to tune the parameter using annealing. The resulting model and learning algorithm has information-theoretic connections to maximum entropy discrimination and the information bottleneck principle. Empirically, we show that the proposed approach significantly outperforms several state-of-the-art baselines, including two recently-proposed neural network approaches, on several real-world datasets. We also provide extended experiments comparing the multinomial likelihood with other commonly used likelihood functions in the latent factor collaborative filtering literature and show favorable results. Finally, we identify the pros and cons of employing a principled Bayesian inference approach and characterize settings where it provides the most significant improvements.


Fraction of Participating Clients One of the characteristics of FL is that data is massively distributed and nodes may have limited communication with the central server as nodes may be offline or have slow connections limiting their communication #REFR .

From Figure 6 we observe WSM has a more significant effect as α decreases with the biggest margin of improvement over vanilla FedAvg occurring when α = 0.01.

As the data becomes increasingly homogeneous, the gap between cross entropy with and without WSM shrinks until the data is i.i.d. and the two methods are equivalent.

From Figure 8 we see that by the time α = 0.5 most of the clients possess all of the classes, albeit in very skewed proportions.

This is also the point at which the performance of FedAvg and FedAvg+WSM become very close.

While WSM continues to offer a performance increase over the entire range of α except for α = 100, we conclude WSM is most advantageous when clients labels distributions are imbalanced.

The direct result of the limited capacity for connection is that only a small number of client nodes may participate in updates at any given time.

These ablations focus on the fraction of clients participating in an update round, for each experiment the fraction of participating clients is constant for all rounds of training and we explore fractions from 1%, where only one client participates in the update, up to 60%.

For the fraction of participating clients p, we observe the largest performance gap between FedAvg+WSM and FedAvg when the number of participating clients is low.

This feature is significant since as explained above, low client participation is a known feature of real world federated learning settings.

We hypothesize the performance gap as a function of client fraction between the FedAvg and FedAvg+WSM is due to the larger impact of local client forgetting when we limit communication capability.

Re-Weighted Softmax Cross-Entropy To Control Forgetting In Federated Learning

Abstract: In Federated Learning, a global model is learned by aggregating model updates computed at a set of independent client nodes, to reduce communication costs multiple gradient steps are performed at each node prior to aggregation. A key challenge in this setting is data heterogeneity across clients resulting in differing local objectives which can lead clients to overly minimize their own local objective, diverging from the global solution. We demonstrate that individual client models experience a catastrophic forgetting with respect to data from other clients and propose an efficient approach that modifies the cross-entropy objective on a per-client basis by re-weighting the softmax logits prior to computing the loss. This approach shields classes outside a client's label set from abrupt representation change and we empirically demonstrate it can alleviate client forgetting and provide consistent improvements to standard federated learning algorithms. Our method is particularly beneficial under the most challenging federated learning settings where data heterogeneity is high and client participation in each round is low.

Communication-Efficient Learning Of Deep Networks From Decentralized Data

Abstract: Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning. We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.


B. Language Models For Clinical Narratives

FlauBERT #REFR : It is based on the original BERT architecture and was trained on a large corpus of the French text.

In this context, we will explore some of the most popular pre-trained BERT-based models for French language processing available from Huggingface.

CamemBERT #OTHEREFR : This is a pre-trained Transformer-based language model designed explicitly for processing French text.

It is based on the Roberta architecture and was trained on a large corpus of French text that was filtered and pre-processed to improve the data quality.

Its pre-training objective is a masked language model, where some input tokens are masked, and the model is trained to predict the missing tokens.

Overall, CamemBERT is a highly effective tool for processing French language text and can be fine-tuned for specific downstream tasks or used for transfer learning in multilingual settings.

It has been shown to perform strongly on several natural language processing tasks in French, including named entity recognition and sentiment analysis.

It also performs well on tasks related to French morphosyntaxes, such as part-of-speech tagging and dependency parsing.

It was trained using a masked language model objective, where a portion of the input tokens are masked, and the model is trained to predict the missing tokens.

FlauBERT is a powerful language model for processing French text that can be fine-tuned for specific downstream tasks.

FrALBERT #OTHEREFR is a Transformer-based language model designed explicitly for text classification tasks in French.

A Small-Scale Switch Transformer And Nlp-Based Model For Clinical Narratives Classification

Abstract: In recent years, Transformer-based models such as the Switch Transformer have achieved remarkable results in natural language processing tasks. However, these models are often too complex and require extensive pre-training, which limits their effectiveness for small clinical text classification tasks with limited data. In this study, we propose a simplified Switch Transformer framework and train it from scratch on a small French clinical text classification dataset at CHU Sainte-Justine hospital. Our results demonstrate that the simplified small-scale Transformer models outperform pre-trained BERT-based models, including DistillBERT, CamemBERT, FlauBERT, and FrALBERT. Additionally, using a mixture of expert mechanisms from the Switch Transformer helps capture diverse patterns; hence, the proposed approach achieves better results than a conventional Transformer with the self-attention mechanism. Finally, our proposed framework achieves an accuracy of 87%, precision at 87%, and recall at 85%, compared to the third-best pre-trained BERT-based model, FlauBERT, which achieved an accuracy of 84%, precision at 84%, and recall at 84%. However, Switch Transformers have limitations, including a generalization gap and sharp minima. We compare it with a multi-layer perceptron neural network for small French clinical narratives classification and show that the latter outperforms all other models.

Flaubert: Unsupervised Language Model Pre-Training For French

Abstract: Language models have become a key step to achieve state-of-the art results in many different Natural Language Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their contextualization at the sentence level. This has been widely demonstrated for English using contextualized representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified evaluation protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research community for further reproducible experiments in French NLP.


Identify Best-Qualified High-Speed Lossy Compressor

In comparison with FXR mode, ZFP's ABS mode features a much higher compression quality on scientific datasets (i.e., higher compression ratio with the same reconstructed data quality), as verified in #REFR .

As shown in the previous analysis, in addition to controlling the data distortion by error bounds, the compression throughput and compression ratio are two critical metrics to consider.

According to the prior literature #OTHEREFR , ZFP and SZx exhibit much higher compression speed than other compressors, including SZ2 #OTHEREFR , SZ3 #OTHEREFR , FPZIP #OTHEREFR , and Auto-SZ #OTHEREFR .

Therefore, we focus on ZFP and SZx in particular and select the best-qualified one for compression-enabled collective communication.

In ZFP, users can set a fixed compression rate in the fixed-rate (FXR) mode or a fixed absolute error bound in the fixed-accuracy (ABS) mode. The two modes have their particular pros and cons.

The advantage of the FXR is that it brings up substantial convenience in some use cases that require knowing the compressed data size in advance.

We briefly analyze the key reason why ZFP's ABS mode has higher compression quality than its FXR mode in the following text.

ZFP divides the dataset into small data blocks during the compression.

Unlike ZFP(ABS) which allows variable compressed size for each block, ZFP(FXR) strictly forces each block to have the same compression ratio, which may inevitably degrade the quality of reconstructed data in turn.

Moreover, the FXR mode cannot control the error bound, which may cause fairly high compression errors on some data points unexpectedly.

In comparison with ZFP, SZx is an ultra-fast error-bounded compressor that was designed driven by the fixed-accuracy mode.

C-Coll: Introducing Error-Bounded Lossy Compression Into Mpi Collectives

Abstract: With the ever-increasing computing power of supercomputers and the growing scale of scientific applications, the efficiency of MPI collective communications turns out to be a critical bottleneck in large-scale distributed and parallel processing. Large message size in MPI collectives is a particularly big concern because it may significantly delay the overall parallel performance. To address this issue, prior research simply applies the off-the-shelf fix-rate lossy compressors in the MPI collectives, leading to suboptimal performance, limited generalizability, and unbounded errors. In this paper, we propose a novel solution, called C-Coll, which leverages errorbounded lossy compression to significantly reduce the message size, resulting in a substantial reduction in communication cost. The key contributions are threefold. (1) We develop two general, optimized lossy-compression-based frameworks for both types of MPI collectives (collective data movement as well as collective computation), based on their particular characteristics. Our framework not only reduces communication cost but also preserves data accuracy. (2) We customize an optimized version based on SZx, an ultra-fast error-bounded lossy compressor, which can meet the specific needs of collective communication. (3) We integrate C-Coll into multiple collectives, such as MPI_Allreduce, MPI_Scatter, and MPI_Bcast, and perform a comprehensive evaluation based on real-world scientific datasets. Experiments show that our solution outperforms the original MPI collectives as well as multiple baselines and related efforts by 3.5-9.7×.

Fraz: A Generic High-Fidelity Fixed-Ratio Lossy Compression Framework For Scientific Floating-Point Data

Abstract: With ever-increasing volumes of scientific floating-point data being produced by high-performance computing applications, significantly reducing scientific floating-point data size is critical, and error-controlled lossy compressors have been developed for years. None of the existing scientific floating-point lossy data compressors, however, support effective fixed-ratio lossy compression. Yet fixed-ratio lossy compression for scientific floating-point data not only compresses to the requested ratio but also respects a user-specified error bound with higher fidelity. In this paper, we present FRaZ: a generic fixed-ratio lossy compression framework respecting user-specified error constraints. The contribution is twofold. (1) We develop an efficient iterative approach to accurately determine the appropriate error settings for different lossy compressors based on target compression ratios. (2) We perform a thorough performance and accuracy evaluation for our proposed fixed-ratio compression framework with multiple state-of-the-art error-controlled lossy compressors, using several real-world scientific floating-point datasets from different domains. Experiments show that FRaZ effectively identifies the optimum error setting in the entire error setting space of any given lossy compressor. While fixed-ratio lossy compression is slower than fixed-error compression, it provides an important new lossy compression technique for users of very large scientific floating-point datasets.


For images, we obtain the faces using MTCNN #REFR and resize them to 160×160. Table 3 shows the results of character detection.

We use the spaCy PoS tagger (https://spacy.io/api/tagger) to obtain the nouns in text and then identify the characters using WordNet #OTHEREFR .

Evaluation Metric We use precision and recall to evaluate textual and visual character detection.

For textual character detection, a predicted character phrase is considered as a correct character detection when the head word of the noun phrase is the same as the gold-standard one.

For visual character detection, a predicted face region is considered as a correct character detection when the bounding box of the face is entirely inside of an annotated body bounding box.

Analysis Overall, recall scores are higher than precision scores for both modalities.

The precision of image-based character detection is only 40.5% and the gap with recall is around 28.6%.

Detecting And Grounding Important Characters In Visual Stories

Abstract: Characters are essential to the plot of any story. Establishing the characters before writing a story can improve the clarity of the plot and the overall flow of the narrative. However, previous work on visual storytelling tends to focus on detecting objects in images and discovering relationships between them. In this approach, characters are not distinguished from other objects when they are fed into the generation pipeline. The result is a coherent sequence of events rather than a charactercentric story. In order to address this limitation, we introduce the VIST-Character dataset, which provides rich charactercentric annotations, including visual and textual co-reference chains and importance ratings for characters. Based on this dataset, we propose two new tasks: important character detection and character grounding in visual stories. For both tasks, we develop simple, unsupervised models based on distributional similarity and pre-trained vision-and-language models. Our new dataset, together with these models, can serve as the foundation for subsequent work on analysing and generating stories from a character-centric perspective.

Joint Face Detection And Alignment Using Multitask Cascaded Convolutional Networks

Abstract: Face detection and alignment in unconstrained environment are challenging due to various poses, illuminations, and occlusions. Recent studies show that deep learning approaches can achieve impressive performance on these two tasks. In this letter, we propose a deep cascaded multitask framework that exploits the inherent correlation between detection and alignment to boost up their performance. In particular, our framework leverages a cascaded architecture with three stages of carefully designed deep convolutional networks to predict face and landmark location in a coarse-to-fine manner. In addition, we propose a new online hard sample mining strategy that further improves the performance in practice. Our method achieves superior accuracy over the state-of-the-art techniques on the challenging face detection dataset and benchmark and WIDER FACE benchmarks for face detection, and annotated facial landmarks in the wild benchmark for face alignment, while keeps real-time performance.


Pairs from the CambridgeLandmarks dataset are computed by encoding images with a pre-trained NetVLAD model #REFR , and fetching the nearest 40 neighbors based on cosine similarity.

At test time, images are rescaled, and the center crop is used for inference, with no further augmentations.

Our model is optimized using Adam, with β 1 = 0.9, β 2 = 0.999 and = 10 −10 and a batch size of 8.

We used an initial learning rate of λ = 10 −4 and a weight decay of 10 −4 and trained for 30 (600) epochs for indoor (outdoor) localization. Our loss is initialized as in #OTHEREFR .

The training pairs for the 7Scenes dataset are generated as in #OTHEREFR .

We further extend this dataset with pairs from #OTHEREFR .

At train time, we sample from the pool of neighbors to generate different pairs.

We provide the training pairs, code, configuration files, and pre-trained models in order to support the reproducibility of our results.

All models were trained on a single NVIDIA RTX A5000 24GB GPU. Implementation Details. We use EfficientNet-B0 #OTHEREFR as our convolutional backbone.

We extract the feature maps F trans and F rot at resolutions of 14 × 14 × 112 and 28 × 28 × 40, respectively, as these endpoints were previously shown to provide useful features for camera pose regression #OTHEREFR (we reconfirm this choice in our ablation study).

We set C h = 512 for the dimension of our projected paired feature maps and their respective positional encoding.

Learning To Localize In Unseen Scenes With Relative Pose Regressors

Abstract: Relative pose regressors (RPRs) localize a camera by estimating its relative translation and rotation to a poselabelled reference. Unlike scene coordinate regression and absolute pose regression methods, which learn absolute scene parameters, RPRs can (theoretically) localize in unseen environments, since they only learn the residual pose between camera pairs. In practice, however, the performance of RPRs is significantly degraded in unseen scenes. In this work, we propose to aggregate paired feature maps into latent codes, instead of operating on global image descriptors, in order to improve the generalization of RPRs. We implement aggregation with concatenation, projection, and attention operations (Transformer Encoders) and learn to regress the relative pose parameters from the resulting latent codes. We further make use of a recently proposed continuous representation of rotation matrices, which alleviates the limitations of the commonly used quaternions. Compared to stateof-the-art RPRs, our model is shown to localize significantly better in unseen environments, across both indoor and outdoor benchmarks, while maintaining competitive performance in seen scenes. We validate our findings and architecture design through multiple ablations. Our code and pretrained models is publicly available 1 .

Netvlad: Cnn Architecture For Weakly Supervised Place Recognition

Abstract: We tackle the problem of large scale visual place recognition, where the task is to quickly and accurately recognize the location of a given query photograph. We present the following four principal contributions. First, we develop a convolutional neural network (CNN) architecture that is trainable in an end-to-end manner directly for the place recognition task. The main component of this architecture, NetVLAD, is a new generalized VLAD layer, inspired by the “Vector of Locally Aggregated Descriptors” image representation commonly used in image retrieval. The layer is readily pluggable into any CNN architecture and amenable to training via backpropagation. Second, we create a new weakly supervised ranking loss, which enables end-to-end learning of the architecture's parameters from images depicting the same places over time downloaded from Google Street View Time Machine. Third, we develop an efficient training procedure which can be applied on very large-scale weakly labelled tasks. Finally, we show that the proposed architecture and training procedure significantly outperform non-learnt image representations and off-the-shelf CNN descriptors on challenging place recognition and image retrieval benchmarks.


Numerical results confirmed the bound's validity and showed that it is even sharper than the state-of-the-art result #REFR .

The main focus of this paper was on linear diffusive problems.

In that setting, and when Assumptions 3.1 to 3.3 are satisfied, we adapted ParaOpt to work for both tracking and terminal-cost objectives, while having generic convergence bounds and a preconditioner that ensures good weak scalability.

We summarize our main contributions in the linear diffusive setting.

• We formulated an extension of ParaOpt to the setting of tracking problems.

• We proved a generalized convergence bound for terminal-cost ParaOpt that is generic in the propagators used (as long as they satisfy Assumptions 3.1 to 3.3).

• We proved a similar bound for the tracking setting and proved that the case of an exact fine and a one-step implicit-Euler coarse propagator guarantees ParaOpt does not diverge.

• Thanks to the generic convergence bound, we were able to study a different type of implicit-Euler coarse propagator for terminal-cost problems and found it improves ParaOpt's convergence.

• We proposed diagonalization-based preconditioners to improve the scaling of ParaOpt.

Analytic results from the ParaDiag literature and numerical tests confirmed that ParaOpt is now a weakly scalable algorithm.

For problems outside the linear diffusive category, ParaOpt is still a very promising method #OTHEREFR , although poorly understood.

Diagonalization-Based Preconditioners And Generalized Convergence Bounds For Paraopt

Abstract: The ParaOpt algorithm was recently introduced as a time-parallel solver for optimalcontrol problems with a terminal-cost objective, and convergence results have been presented for the linear diffusive case with implicit-Euler time integrators. We reformulate ParaOpt for tracking problems and provide generalized convergence analyses for both objectives. We focus on linear diffusive equations and prove convergence bounds that are generic in the time integrators used. For large problem dimensions, ParaOpt's performance depends crucially on having a good preconditioner to solve the arising linear systems. For the case where ParaOpt's cheap, coarse-grained propagator is linear, we introduce diagonalization-based preconditioners, inspired by recent advances in the ParaDiag family of methods. These preconditioners not only lead to a weakly-scalable ParaOpt version, but are themselves invertible in parallel, making maximal use of available concurrency. They have proven convergence properties in the linear diffusive case that are generic in the time discretization used, similarly to our ParaOpt results. Numerical results confirm that the iteration count of the iterative solvers used for ParaOpt's linear systems becomes constant in the limit of an increasing processor count. The paper is accompanied by a sequential Matlab implementation.

Paraopt: A Parareal Algorithm For Optimality Systems

Abstract: The time parallel solution of optimality systems arising in PDE constraint optimization could be achieved by simply applying any time parallel algorithm, such as Parareal, to solve the forward and backward evolution problems arising in the optimization loop. We propose here a different strategy by devising directly a new time parallel algorithm, which we call ParaOpt, for the coupled forward and backward non-linear partial differential equations. ParaOpt is inspired by the Parareal algorithm for evolution equations, and thus is automatically a two-level method. We provide a detailed convergence analysis for the case of linear parabolic PDE constraints. We illustrate the performance of ParaOpt with numerical experiments both for linear and nonlinear optimality systems.


While our research shares some similarities with previous studies #REFR [26], we believe that our loss function can serve as a significant complement to address different scenarios.

But only if the training process is sufficiently, it can be beneficial to the test robustness.

In this paper, we propose a distribution-restrained softmax loss for adversarial robustness.

Based on adversarial attack tests, we find that pre-and post-attack softmax are highly correlated.

According to the relationship, we come up with a loss function to suppress the distribution diversity of softmax.

By testing of this new loss function, we suggest that our method can improve the model robustness.

Further comparative discussion will be conducted later.

Distribution-Restrained Softmax Loss For The Model Robustness

Abstract: Recently, the robustness of deep learning models has received widespread attention, and various methods for improving model robustness have been proposed, including adversarial training, model architecture modification, design of loss functions, certified defenses, and so on. However, the principle of the robustness to attacks is still not fully understood, also the related research is still not sufficient. Here, we have identified a significant factor that affects the robustness of models: the distribution characteristics of softmax values for non-real label samples. We found that the results after an attack are highly correlated with the distribution characteristics, and thus we proposed a loss function to suppress the distribution diversity of softmax. A large number of experiments have shown that our method can improve robustness without significant time consumption.

Distillation As A Defense To Adversarial Perturbations Against Deep Neural Networks

Abstract: Deep learning algorithms have been shown to perform extremely well on many classical machine learning problems. However, recent studies have shown that deep learning, like other machine learning techniques, is vulnerable to adversarial samples: inputs crafted to force a deep neural network (DNN) to provide adversary-selected outputs. Such attacks can seriously undermine the security of the system supported by the DNN, sometimes with devastating consequences. For example, autonomous vehicles can be crashed, illicit or illegal content can bypass content filters, or biometric authentication systems can be manipulated to allow improper access. In this work, we introduce a defensive mechanism called defensive distillation to reduce the effectiveness of adversarial samples on DNNs. We analytically investigate the generalizability and robustness properties granted by the use of defensive distillation when training DNNs. We also empirically study the effectiveness of our defense mechanisms on two DNNs placed in adversarial settings. The study shows that defensive distillation can reduce effectiveness of sample creation from 95% to less than 0.5% on a studied DNN. Such dramatic gains can be explained by the fact that distillation leads gradients used in adversarial sample creation to be reduced by a factor of 10^30. We also find that distillation increases the average minimum number of features that need to be modified to create adversarial samples by about 800% on one of the DNNs we tested.


One of the main challenges for resilient state estimation is that many of the proposed algorithms are combinatorial in nature #REFR .

To solve the resilient state estimation problem, there have been techniques developed based on fault detection and diagnosis #OTHEREFR , #OTHEREFR , as in #OTHEREFR , #OTHEREFR ; methods motivated from compressed sensing #OTHEREFR , #OTHEREFR , as in #OTHEREFR ; and observer-based methods as in #OTHEREFR - #OTHEREFR .

Extensions have been presented for decentralized design and distributed operation #OTHEREFR - #OTHEREFR , nonlinear systems #OTHEREFR - #OTHEREFR , and complexity analysis #OTHEREFR .

Hence, the algorithms cannot solve the problem in polynomial time in general, and substantial amount of computational or storage resources may be required, especially when the number of sensors is large.

For instance, to identify and isolate/exclude q unknown compromised sensors out of p sensors (q < p), methods in #OTHEREFR , #OTHEREFR construct not less than p q observers for the computation, and methods in #OTHEREFR , #OTHEREFR consider p q cases to search out the un-compromised sensors.

Reducing the complexity of resilient state estimation consequently has been major interest.

Many sufficient conditions that allow complexity reduction have been found, specifically for linear systems.

For example, a condition for converting the problem into a convex optimization problem is introduced in #OTHEREFR , and an assumption for making use of gradient decent algorithms is found in #OTHEREFR .

Abstract: A resilient state estimation scheme for uniformly observable nonlinear systems, based on a method for local identification of sensor attacks, is presented. The estimation problem is combinatorial in nature, and so many methods require substantial computational and storage resources as the number of sensors increases. To reduce the complexity, the proposed method performs the attack identification with local subsets of the measurements, not with the set of all measurements. A condition for nonlinear attack identification is introduced as a relaxed version of existing redundant observability condition. It is shown that an attack identification can be performed even when the state cannot be recovered from the measurements. As a result, although a portion of measurements are compromised, they can be locally identified and excluded from the state estimation, and thus the true state can be recovered. Simulation results demonstrate the effectiveness of the proposed scheme.

Attack Detection And Identification In Cyber-Physical Systems

Abstract: Cyber-physical systems integrate computation, communication, and physical capabilities to interact with the physical world and humans. Besides failures of components, cyber-physical systems are prone to malignant attacks, and specific analysis tools as well as monitoring mechanisms need to be developed to enforce system security and reliability. This paper proposes a unified framework to analyze the resilience of cyber-physical systems against attacks cast by an omniscient adversary. We model cyber-physical systems as linear descriptor systems, and attacks as exogenous unknown inputs. Despite its simplicity, our model captures various real-world cyber-physical systems, and it includes and generalizes many prototypical attacks, including stealth, (dynamic) false-data injection and replay attacks. First, we characterize fundamental limitations of static, dynamic, and active monitors for attack detection and identification. Second, we provide constructive algebraic conditions to cast undetectable and unidentifiable attacks. Third, by using the system interconnection structure, we describe graph-theoretic conditions for the existence of undetectable and unidentifiable attacks. Finally, we validate our findings through some illustrative examples with different cyber-physical systems, such as a municipal water supply network and two electrical power grids.


To address this gap, we will incorporate explainable AI techniques #REFR into our future work to provide a human-understandable interpretation of the predictions.

Our study showcases the potential of a tree-distillation multi-task learning framework from computational and clinical perspectives.

This approach combines the benefits of a tree model and multi-task learning, resulting in competitive performance on transplant EHR data.

The dependable outputs from CoD-MTL indicate its potential as a promising clinical decision support tool for physicians involved in organ transplant procedures, as demonstrated by our case study focusing on liver transplants.

Additionally, we need to address some limitations and identify corresponding solutions for future improvement of our CoD-MTL framework.

A critical constraint of our current model is its lack of interpretability, as the inability to explain predictions may impede its deployment in organ transplant scenarios.

Another limitation is the failure to consider fairness in our current framework.

Equity is an ethical goal that clinical decision support systems should aim to achieve #OTHEREFR .

Therefore, our future work will place a strong emphasis on integrating fairness constraints within our prediction framework to ensure it is suitable for multiple outcomes in organ transplants.

The proposed framework has the potential to extend beyond our focus on organ transplants and can be applied to other medical fields that use multi-task learning.

For instance, the prediction of complications and the length of stay in the ICU #OTHEREFR may benefit from our CoD-MTL framework, subject to future refinement.

Multi-Task Learning For Post-Transplant Cause Of Death Analysis: A Case Study On Liver Transplant

Abstract: Organ transplant is the essential treatment method for some end-stage diseases, such as liver failure. Analyzing the post-transplant cause of death (CoD) after organ transplant provides a powerful tool for clinical decision making, including personalized treatment and organ allocation. However, traditional methods like Model for End-stage Liver Disease (MELD) score and conventional machine learning (ML) methods are limited in CoD analysis due to two major data and model-related challenges. To address this, we propose a novel framework called CoD-MTL leveraging multi-task learning to model the semantic relationships between various CoD prediction tasks jointly. Specifically, we develop a novel tree distillation strategy for multi-task learning, which combines the strength of both the tree model and multi-task learning. Experimental results are presented to show the precise and reliable CoD predictions of our framework. A case study is conducted to demonstrate the clinical importance of our method in the liver transplant.

A Survey On Explainable Artificial Intelligence (Xai): Towards Medical Xai

Abstract: Recently, artificial intelligence, especially machine learning has demonstrated remarkable performances in many tasks, from image processing to natural language processing, especially with the advent of deep learning. Along with research progress, machine learning has encroached into many different fields and disciplines. Some of them, such as the medical field, require high level of accountability, and thus transparency, which means we need to be able to explain machine decisions, predictions and justify their reliability. This requires greater interpretability, which often means we need to understand the mechanism underlying the algorithms. Unfortunately, the black-box nature of the deep learning is still unresolved, and many machine decisions are still poorly understood. We provide a review on interpretabilities suggested by different research works and categorize them, with the intention of providing alternative perspective that is hopefully more tractable for future adoption of interpretability standard. We explore further into interpretability in the medical field, illustrating the complexity of interpretability issue.


Qudit Teleportation Through Quantum Switch

Fidelity starts increasing after the threshold even as noise affecting the channel increases, thereby corroborating the results obtained in #REFR .

When p = q = 0.5 (like a 50% erasure channel), no quantum information can be sent through any classical trajectory traversing the channels A and B #OTHEREFR .

Hence, achievable fidelity increases with the dimension of the quantum particle used for teleportation.

Increased dimension refers to an enlarged Hilbert space and a larger Hilbert space offers the advantage of communicating with larger information capacity and higher noise resilience than traditional qubit-based quantum systems. It is evident from Fig.

6 , average fidelityF QS decreases as p and q increase (i.e.

increase in error probabilities), up to a certain threshold like p = q = 1/2 for the case of d = 2.

For the case when p = q → 0.5 (the limiting condition of zero-capacity quantum channel), teleportation of qubit over a quantum switch offers a fidelity of approaching 0.5, i.e.F QS d=2 → 0.5.

While as we increase d to 5, fidelity approaches 0.65.

Therefore, entanglement distribution of a qudit over a quantum switch provides a remarkable gain in fidelity of the teleported qudit at Bob's side with increase in both noise plaguing the communication channel and dimensionality of the quantum particle.

For any scenario of quantum teleportation, the EPR pair while being distributed between transmitter and receiver may lose its coherence, and become a mixed state due to interaction with the environment and noise introduced by the quantum channel.

Entanglement Distribution And Quantum Teleportation In Higher Dimension Over The Superposition Of Causal Orders Of Quantum Channels *

Abstract: Multiple photonic degrees of freedom can be explored to generate high-dimensional quantum states; commonly referred to as 'qudits'. Qudits offer several advantages for quantum communications, including higher information capacity, noise resilience and data throughput, and lower information loss over different propagation mediums (free space, optical fibre, underwater) as compared to conventional qubits based communication system. However, qudits have been little exploited in literature, owing to their difficulty in transmission and detection. In this paper, for the first time, we develop and formulate the theoretical framework for transmission of classical information through entanglement distribution of qudits over two quantum channels in superposition of alternative causal order. For the first time we i) engineer quantum switch operation for 2-qudit systems and ii) formulate theoretical system model for entanglement distribution of qudits via quantum switch. Results show that entanglement distribution of a qudit provides a considerable gain in fidelity even with increase in noise.

Quantum Switch For The Quantum Internet: Noiseless Communications Through Noisy Channels

Abstract: Counter-intuitively, quantum mechanics enables quantum particles to propagate simultaneously among multiple space-time trajectories. Hence, a quantum information carrier can travel through different communication channels in a quantum superposition of different orders, so that the relative causal order of the communication channels becomes indefinite. This is realized by utilizing a quantum device known as quantum switch. In this paper, we investigate, from a communications engineering perspective, the use of the quantum switch within the quantum teleportation process, one of the key functionalities of the Quantum Internet. Specifically, a theoretical analysis is conducted to quantify the performance gain that can be achieved by employing a quantum switch for the entanglement distribution process within the quantum teleportation, with respect to the case of absence of the quantum switch. The analysis reveals that, by utilizing the quantum switch, the quantum teleportation is heralded as a noiseless communication process with a probability that, remarkably and counter-intuitively, increases with the noise levels affecting the communication channels considered in the indefinite-order combination.


For instance, SPIN #REFR incorporates the optimization process into the regression framework, allowing better-optimized 3D results as supervision for the network.

Optimization-based approaches concentrate on the optimization algorithms to fit parametric models (e.g., SMPL #OTHEREFR ) based on 2D observations such as keypoints and silhouettes.

These methods have been criticized for having long optimization processes and high sensitivity to initialization #OTHEREFR .

Therefore, regression-based approaches have become a more popular paradigm in recent years, as they can directly and fast regress parameters from image features.

HMR #OTHEREFR is a milestone work that introduces reprojection loss of keypoints as weak supervision, enabling training regression models on in-the-wild datasets.

Recent work continues to explore more advanced weak supervision loss, such as introducing fine-grained correspondences #OTHEREFR , providing pseudo 3D ground truth #OTHEREFR , and improving reprojection loss #OTHEREFR .

SPEC #OTHEREFR estimates the perspective camera from a single image and utilizes it to improve reprojection loss.

CLIFF #OTHEREFR proposes considering the box information and computing the reprojection loss from the original images, which facilitates predicting global rotations.

However, all these methods employ global representation for regression, making them sensitive to occlusion and partially visible humans.

Handling occlusion is challenging yet crucial for human shape and pose estimation.

Bopr: Body-Aware Part Regressor For Human Shape And Pose Estimation

Abstract: Figure 1: Illustration of our method's main superiority over existing approaches. When given a challenging input (a), PARE (b) [16] and CLIFF (c) [20] struggle to accurately estimate the pose of the forearm due to occlusion and depth ambiguity. Our method (d), on the other hand, tackles these challenges by exploiting body-part attention dependency in perpart estimation rather than relying on independent part regressors or global regression. This approach enables our method to more effectively reason about the relative spatial relationship between different body parts and the body as a whole, leading to improved occlusion handling and reduced depth ambiguity.

Learning To Reconstruct 3D Human Pose And Shape Via Model-Fitting In The Loop

Abstract: Model-based human pose estimation is currently approached through two different paradigms. Optimization-based methods fit a parametric body model to 2D observations in an iterative manner, leading to accurate image-model alignments, but are often slow and sensitive to the initialization. In contrast, regression-based methods, that use a deep network to directly estimate the model parameters from pixels, tend to provide reasonable, but not pixel accurate, results while requiring huge amounts of supervision. In this work, instead of investigating which approach is better, our key insight is that the two paradigms can form a strong collaboration. A reasonable, directly regressed estimate from the network can initialize the iterative optimization making the fitting faster and more accurate. Similarly, a pixel accurate fit from iterative optimization can act as strong supervision for the network. This is the core of our proposed approach SPIN (SMPL oPtimization IN the loop). The deep network initializes an iterative optimization routine that fits the body model to 2D joints within the training loop, and the fitted estimate is subsequently used to supervise the network. Our approach is self-improving by nature, since better network estimates can lead the optimization to better solutions, while more accurate optimization fits provide better supervision for the network. We demonstrate the effectiveness of our approach in different settings, where 3D ground truth is scarce, or not available, and we consistently outperform the state-of-the-art model-based pose estimation approaches by significant margins. The project website with videos, results, and code can be found at this https URL.


ResNet18: This model introduced by He et al #REFR , consists of 18-layers with skip connections and batch normalization.

Using our framework, we evaluate the following off-theshelf pretrained models:

We refer to ResNet18 here as, the widely used version of this model that is trained on the Imagenet-1k classification challenge #OTHEREFR .

ResNet50: Similar to ResNet18, but this model has 50 layers #OTHEREFR .

We refer to ResNet50 here as specifically the model trained on the Imagenet-1k classification challenge #OTHEREFR .

BiT-M-R50x1: This model also leverages the ResNet50 model architecture but is trained on the full Imagenet dataset containing 21k classes #OTHEREFR .

The original full Imagenet-21k dataset might not be available #OTHEREFR , checkpoints for this model are publicly available.

Variation Of Gender Biases In Visual Recognition Models Before And After Finetuning

Abstract: We introduce a framework to measure how biases change before and after fine-tuning a large scale visual recognition model for a downstream task. Deep learning models trained on increasing amounts of data are known to encode societal biases. Many computer vision systems today rely on models typically pretrained on large scale datasets. While bias mitigation techniques have been developed for tuning models for downstream tasks, it is currently unclear what are the effects of biases already encoded in a pretrained model. Our framework incorporates sets of canonical images representing individual and pairs of concepts to highlight changes in biases for an array of off-the-shelf pretrained models across model sizes, dataset sizes, and training objectives. Through our analyses, we find that (1) supervised models trained on datasets such as ImageNet-21k are more likely to retain their pretraining biases regardless of the target dataset compared to self-supervised models. We also find that (2) models finetuned on larger scale datasets are more likely to introduce new biased associations. Our results also suggest that (3) biases can transfer to finetuned models and the finetuning objective and dataset can impact the extent of transferred biases.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


It takes advantage of automatic differentiation #REFR in formulating a physical loss in the loss function along with data loss.

Modelling solutions that leverage the strengths of theory-guided as well as data-driven models have the potential to reduce data needs, increase robustness, and effectively use theoretical and practical knowledge of the system.

To investigate model architectures, balancing the strengths of both theory-based models and data-driven models, this work explores the application of physics-informed neural networks (PINNs) to a diesel internal combustion engine model for the purposes of simultaneous parameter and state estimation.

The physical portion is based on the mean value model of a diesel engine with a variable geometry turbocharger (VGT), and exhaust gas recirculation (EGR) proposed by Wahlström and Eriksson #OTHEREFR .

Physics Inform Neural Networks (PINNs) [2-4, 7, 8] is a new method of training neural networks, which takes into account the physics of a problem while evaluating the parameters of the neural network.

The method is suitable for both evaluation of the solution of PDF (forward problem) and the data-driven identification of parameters of PDF (inverse problem).

In the present study, we formulate a PINN model for the data-driven identification of parameters and prediction of dynamics of system variables of a diesel engine.

Specifically, we are interested in estimating unknown parameters and states when we know a few state variables from field data.

The dynamics of the state variables of the mean value engine #OTHEREFR are described by first-order differential equations.

We will utilize these equations in the formulation of the physics-informed loss function.

The unknown parameters are considered trainable and updated in the training process along with the neural network parameters.

Physics-Informed Neural Networks For Predicting Gas Flow Dynamics And Unknown Parameters In Diesel Engines

Abstract: This paper presents a physics-informed neural network (PINN) approach for monitoring the health of diesel engines. The aim is to evaluate the engine dynamics, identify unknown parameters in a "mean value" model, and anticipate maintenance requirements. The PINN model is applied to diesel engines with a variable-geometry turbocharger and exhaust gas recirculation, using measurement data of selected state variables. The results demonstrate the ability of the PINN model to predict simultaneously both unknown parameters and dynamics accurately with both clean and noisy data, and the importance of the self-adaptive weight in the loss function for faster convergence. The input data for these simulations are derived from actual engine running conditions, while the outputs are simulated data, making this a practical case study of PINN's ability to predict real-world dynamical systems. The mean value model of the diesel engine incorporates empirical formulae to represent certain states, but these formulae may not be generalizable to other engines. To address this, the study considers the use of deep neural networks (DNNs) in addition to the PINN model. The DNNs are trained using laboratory test data and are used to model the engine-specific empirical formulae in the mean value model, allowing for a more flexible and adaptive representation of the engine's states. In other words, the mean value model uses both the PINN model and the DNNs to represent the engine's states, with the PINN providing a physics-based understanding of the engine's overall dynamics and the DNNs offering a more engine-specific and adaptive representation of the empirical formulae. By combining these two approaches, the study aims to offer a comprehensive and versatile approach to monitoring the health and performance of diesel engines.

Automatic Differentiation In Machine Learning: A Survey

Abstract: Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply “auto-diff”, is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational fluid dynamics, atmospheric sciences, and engineering design optimization. Until ::: very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other’s results. Despite its ::: relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names “dynamic computational ::: graphs” and “differentiable programming”. We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main imple- ::: mentation techniques. By precisely defining the main differentiation techniques and their interrelationships, we aim to bring clarity to the usage of the terms “autodiff”, “automatic differentiation”, and “symbolic differentiation” as these are encountered more and more in machine learning settings.


Capturing dense multimodal interactions: Early fusion of histology and pathway tokens can be done with a Transformer that uses self-attention to capture pairwise similarities between all tokens #REFR .

Instead, we propose tokenizing genes according to established biological pathways #OTHEREFR - #OTHEREFR .

Pathways are gene sets with known interactions that relate to specific cellular functions, such as the TGF-β signaling cascade, which contributes to the epithelial-mesenchymal transition in breast cancer #OTHEREFR .

Compared to coarse sets (e.g., N P = 6 #OTHEREFR ), pathwaybased gene grouping can yield hundreds to thousands of tokens that represent unique molecular processes (N P = 331 in our work), which we hypothesize are more suitable representations for multimodal fusion and alignment with fine-grained pathology features.

In addition, as pathways represent unique cellular functions, they constitute appropriate basic reasoning units suitable for interpretability (see Fig. 1 ).

However, modeling pairwise interactions between large sets of histology patch tokens (e.g., N H = 15, 000) and pathway tokens (N P = 331) poses scalability challenges for fusion and alignment.

Due to the quadratic complexity of the Transformer attention with respect to the number of tokens, modeling all possible interactions imposes substantial computational and memory requirements.

To tackle this issue, we introduce a new unified, memory-efficient attention mechanism that can successfully model patch-to-pathway, pathway-to-patch, and pathway-to-pathway interactions.

Modeling these three forms of interaction is achieved by the following: (1) designing the queries, keys, and values to share parameters across token types #OTHEREFR , #OTHEREFR , and (2) simplifying the attention layer to ignore patch-to-patch interactions, which we find through experimentation to be not as effective for survival analysis.

Adapting Transformer attention in such a manner leads to significant performance increase over prior works, as well as new applications for model interpretability in multimodal CPATH, especially by understanding interactions between pathways and morphological patterns.

Modeling Dense Multimodal Interactions Between Biological Pathways And Histology For Survival Prediction

Abstract: Integrating whole-slide images (WSIs) and bulk transcriptomics for predicting patient survival can improve our understanding of patient prognosis. However, this multimodal task is particularly challenging due to the different nature of these data: WSIs represent a very high-dimensional spatial description of a tumor, while bulk transcriptomics represent a global description of gene expression levels within that tumor. In this context, our work aims to address two key challenges: (1) how can we tokenize transcriptomics in a semantically meaningful and interpretable way?, and (2) how can we capture dense multimodal interactions between these two modalities? Specifically, we propose to learn biological pathway tokens from transcriptomics that can encode specific cellular functions. Together with histology patch tokens that encode the different morphological patterns in the WSI, we argue that they form appropriate reasoning units for downstream interpretability analyses. We propose fusing both modalities using a memory-efficient multimodal Transformer that can model interactions between pathway and histology patch tokens. Our proposed model, SURVPATH, achieves stateof-the-art performance when evaluated against both unimodal and multimodal baselines on five datasets from The Cancer Genome Atlas. Our interpretability framework identifies key multimodal prognostic factors, and, as such, can provide valuable insights into the interaction between genotype and phenotype, enabling a deeper understanding of the underlying biological mechanisms at play. We make our code public at https://github.com/ajv012/SurvPath.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

dense multimodal interactions
Transformer


We extract these features using SuperPoint #REFR , a single-pass convolutional neural network trained using homographic adaptation, a self-supervised consistency method.

FBE greedily selects the nearest frontier in the map to navigate to.

Upon reaching that frontier, the process repeats with the selection of the next nearest frontier.

This policy enables an efficient expansion of coverage with demonstrated effectiveness in both simulation and reality #OTHEREFR . Goal Instance Re-Identification.

We employ a keypointbased re-identification method that performs binary classification conditioned on the goal image and egocentric image.

First, we extract the pixel-wise (x, y) location of keypoints K * ∈ R n×2 in the goal image and their associated vector descriptions V * ∈ R n×256 .

We repeat this for the egocentric image, resulting in K t ∈ R m×2 and V t ∈ R m×256 . Concretely:

We then compute correspondences between (K * , V * ) and

(K t , V t ), resulting in a matched subset (K * ∈ R w×2 ,K t ∈ R w×2

) such that the i th keypoint ofK * corresponds to the i th keypoint ofK t with w ≤ min(n, m).

Navigating To Objects Specified By Images

Abstract: Images are a convenient way to specify which particular object instance an embodied agent should navigate to. Solving this task requires semantic visual reasoning and exploration of unknown environments. We present a system that can perform this task in both simulation and the real world. Our modular method solves sub-tasks of exploration, goal instance re-identification, goal localization, and local navigation. We re-identify the goal instance in egocentric vision using feature-matching and localize the goal instance by projecting matched features to a map. Each sub-task is solved using off-the-shelf components requiring zero finetuning. On the HM3D InstanceImageNav benchmark, this system outperforms a baseline end-to-end RL policy 7x and a state-of-the-art ImageNav model 2.3x (56% vs. 25% success). We deploy this system to a mobile robot platform and demonstrate effective real-world performance, achieving an 88% success rate across a home and an office environment.

Superpoint: Self-Supervised Interest Point Detection And Description

Abstract: This paper presents a self-supervised framework for training interest point detectors and descriptors suitable for a large number of multiple-view geometry problems in computer vision. As opposed to patch-based neural networks, our fully-convolutional model operates on full-sized images and jointly computes pixel-level interest point locations and associated descriptors in one forward pass. We introduce Homographic Adaptation, a multi-scale, multi-homography approach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g., synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able to repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other traditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches when compared to LIFT, SIFT and ORB.

self-supervised consistency method
homographic adaptation