However, likelihood-based regression strategies have been proven to deal with numbers having high variance in their magnitudes #REFR .

Numerical Reasoning in Language Modeling While PLM has been widely used in general natural language processing tasks #OTHEREFR , handling numerals has recently started to gain focus in the NLP domain #OTHEREFR .

Various tasks including probing numerical knowledge #OTHEREFR and probing numerical embedding #OTHEREFR showed the possibility of PLMs achieving numeracy.

While PLMs have been known to learn the scalar magnitude of objects #OTHEREFR , some domain-specific datasets may pose challenges to its learning on regression tasks due to their distribution characteristics.

Previous work tried a traditional linear regression architecture with simple scalar value estimation #OTHEREFR .

Recently, DExp method has been introduced and reported to achieve state-of-the-art performance compared to other likelihood functions-based regression approaches such as Gaussian Mixture Model, Transformed Laplace, and LogLaplace #OTHEREFR .

For our KitchenScale model, we adopted the DExp approach in our ingredient quantity regression task as it has shown success in previous works featuring numeral values with highly varying scales #OTHEREFR .

Kitchenscale: Learning To Predict Ingredient Quantities From Recipe Contexts

Abstract: Determining proper quantities for ingredients is an essential part of cooking practice from the perspective of enriching tastiness and promoting healthiness. We introduce KitchenScale, a fine-tuned Pre-trained Language Model (PLM) that predicts a target ingredient's quantity and measurement unit given its recipe context. To effectively train our KitchenScale model, we formulate an ingredient quantity prediction task that consists of three sub-tasks which are ingredient measurement type classification, unit classification, and quantity regression task. Furthermore, we utilized transfer learning of cooking knowledge from recipe texts to PLMs. We adopted the Discrete Latent Exponent (DExp) method to cope with high variance of numerical scales in recipe corpora. Experiments with our newly constructed dataset and recommendation examples demonstrate KitchenScale's understanding of various recipe contexts and generalizability in predicting ingredient quantities. We implemented a web application for KitchenScale to demonstrate its functionality in recommending ingredient quantities expressed in numerals (e.g., 2) with units (e.g., ounce).

Numeracy For Language Models: Evaluating And Improving Their Ability To Predict Numbers

Abstract: Numeracy is the ability to understand and work with numbers. It is a necessary skill for composing and understanding documents in clinical, scientific, and other technical domains. In this paper, we explore different strategies for modelling numerals with language models, such as memorisation and digit-by-digit composition, and propose a novel neural architecture that uses a continuous probability density function to model numerals from an open vocabulary. Our evaluation on clinical and scientific datasets shows that using hierarchical models to distinguish numerals from words improves a perplexity metric on the subset of numerals by 2 and 4 orders of magnitude, respectively, over non-hierarchical models. A combination of strategies can further improve perplexity. Our continuous probability density function model reduces mean absolute percentage errors by 18% and 54% in comparison to the second best strategy for each dataset, respectively.


Other works use centroidal dynamics to inform a fullorder dynamics model solve with CIO #REFR .

A different approach is to increase the accuracy of the solution, which allows for a coarser discretization of the trajectory.

This has been achieved through variational integrators #OTHEREFR and higher-order collocation methods #OTHEREFR , although they still remain quite slow and can struggle with convergence.

Some researchers have blended these approaches by combining solutions of relaxed and exact problems.

One example is to first fix the complementarity constraint relaxation to a large value and then iteratively reduce it #OTHEREFR .

Another approach first solves the problem with first-order collocation and a large relaxation and then use this solution to warm start a problem with third-order collocation and a small relaxation #OTHEREFR .

These methods ensure that the final result will meet the physical constraints of the problem and result in overall decreases in solve times, but still struggle with convergence due to the full-order complexity of the dynamics in combination with the complementarity constraints.

More recently, MIP was applied to an iterative convex approximation of centroidal dynamics and full kinematics, achieving impressive convergence rates #OTHEREFR .

Staged Contact Optimization: Combining Contact-Implicit And Multi-Phase Hybrid Trajectory Optimization

Abstract: Trajectory optimization problems for legged robots are commonly formulated with fixed contact schedules. These multi-phase Hybrid Trajectory Optimization (HTO) methods result in locally optimal trajectories, but the result depends heavily upon the predefined contact mode sequence. Contact-Implicit Optimization (CIO) offers a potential solution to this issue by allowing the contact mode to be determined throughout the trajectory by the optimization solver. However, CIO suffers from long solve times and convergence issues. This work combines the benefits of these two methods into one algorithm: Staged Contact Optimization (SCO). SCO tightens constraints on contact in stages, eventually fixing them to allow robust and fast convergence to a feasible solution. Results on a planar biped and spatial quadruped demonstrate speed and optimality improvements over CIO and HTO. These properties make SCO well suited for offline trajectory generation or as an effective tool for exploring the dynamic capabilities of a robot.

Hierarchical Planning Of Dynamic Movements Without Scheduled Contact Sequences

Abstract: Most animal and human locomotion behaviors for solving complex tasks involve dynamic motions and rich contact interaction. In fact, complex maneuvers need to consider dynamic movement and contact events at the same time. We present a hierarchical trajectory optimization approach for planning dynamic movements with unscheduled contact sequences. We compute whole-body motions that achieve goals that cannot be reached in a kinematic fashion. First, we find a feasible CoM motion according to the centroidal dynamics of the robot. Then, we refine the solution by applying the robot's full-dynamics model, where the feasible CoM trajectory is used as a warm-start point. To accomplish the unscheduled contact behavior, we use complementarity constraints to describe the contact model, i.e. environment geometry and non-sliding active contacts. Both optimization phases are posed as Mathematical Program with Complementarity Constraints (MPCC). Experimental trials demonstrate the performance of our planning approach in a set of challenging tasks.


A disadvantage of our approach is that it requires designing controllers per time step as in #REFR . This is the trade-off between performance and engineering effort.

The feedback gain is obtained by solving nonlinear optimization programs.

In the experiment section, we compare our method with other probabilistic control methods that could track the nominal trajectory, and we show that our approach outperforms other methods.

Apart from better performance, our approach has the following advantages.

First, the approach applies to a general class of stochastic nonlinear systems, more precisely, systems with mixed-trigonometricpolynomial dynamics, which covers most robotic systems, and the approach applies to arbitrary probabilistic uncertainties, not necessarily Gaussian distributions.

Second, our approach leaves room for nonlinear state feedback control, not necessarily linear state feedback control.

Note that we assume the open-loop nominal trajectory is given and we design feedback controllers to track the nominal trajectory.

There are many trajectory optimization methods that could compute the nominal trajectory #OTHEREFR , and how the nominal trajectory is obtained is not the focus of this paper.

This assumption is different from most nonlinear covariance steering literature #OTHEREFR - #OTHEREFR , especially methods similar to DDP, where they design a nominal trajectory and feedback controllers together.

We believe that separately designing the nominal trajectory and feedback controllers can be beneficial in many cases.

For instance, the trajectory optimization alone could incorporate more constraints, such as obstacle avoidance, chance constraints, tube safety, and so on, while it is not easy to incorporate more constraints into an algorithm that simultaneously generate nominal trajectories and feedback controllers, especially when dealing with stochastic nonlinear systems.

Non-Gaussian Uncertainty Minimization Based Control Of Stochastic Nonlinear Robotic Systems

Abstract: In this paper, we consider the closed-loop control problem of nonlinear robotic systems in the presence of probabilistic uncertainties and disturbances. More precisely, we design a state feedback controller that minimizes deviations of the states of the system from the nominal state trajectories due to uncertainties and disturbances. Existing approaches to address the control problem of probabilistic systems are limited to particular classes of uncertainties and systems such as Gaussian uncertainties and processes and linearized systems. We present an approach that deals with nonlinear dynamics models and arbitrary known probabilistic uncertainties. We formulate the controller design problem as an optimization problem in terms of statistics of the probability distributions including moments and characteristic functions. In particular, in the provided optimization problem, we use moments and characteristic functions to propagate uncertainties throughout the nonlinear motion model of robotic systems. In order to reduce the tracking deviations, we minimize the uncertainty of the probabilistic states around the nominal trajectory by minimizing the trace and the determinant of the covariance matrix of the probabilistic states. To obtain the state feedback gains, we solve deterministic optimization problems in terms of moments, characteristic functions, and state feedback gains using off-the-shelf interior-point optimization solvers. To illustrate the performance of the proposed method, we compare our method with existing probabilistic control methods.

Sequential Chance Optimization For Flow-Tube Based Control Of Probabilistic Nonlinear Systems

Abstract: In this paper, we address the problem of closed-loop control of nonlinear dynamical systems subjected to probabilistic uncertainties. More precisely, we design time-varying polynomial feedback controllers to follow the given nominal trajectory and also, for safety purposes, remain in the tube around the nominal trajectory, despite all uncertainties. We formulate this problem as a chance optimization problem where we maximize the probability of achieving control objectives. To address control problems with long planning horizons, we formulate the single large chance optimization problem as a sequence of smaller chance optimization problems. To solve the obtained chance optimization problems, we leverage the theory of measures and moments and obtain convex relaxations in the form of semidefinite programs. We provide numerical examples on stabilizing controller design and motion planning of uncertain nonlinear systems to illustrate the performance of the proposed approach.


In line with this, recent research specifically on the personality of developers has not found any evidence of variation over time #REFR .

Given the low number of items per construct, these constitute acceptable values #OTHEREFR .

We measured narcissism using the short version of the Narcissistic Personality Inventory (NPI-16) #OTHEREFR .

Cronbach's α was .70, constituting an acceptable value #OTHEREFR and even being slightly higher than that obtained in the work that developed the measure in the first place #OTHEREFR .

As we measure personality after the analyzed commits were made, this might potentially raise concerns about an unclear direction of causality in our analysis.

However, this is unlikely to be a problem as personality is widely considered as stable across adult life #OTHEREFR , #OTHEREFR and the sampled developers were all between 24 and 65 years of age.

To measure developers' age at the time of each commit, we asked developers to simply provide their age in years.

We then subtracted the difference between 2022 and the year in which the focal commit was made from the provided age.

The Type To Take Out A Loan? A Study Of Developer Personality And Technical Debt

Abstract: Background: Technical debt (TD) has been widely discussed in software engineering research, and there is an emerging literature linking it to developer characteristics. However, developer personality has not yet been studied in this context. Aims and Method: We explore the relationship between various personality traits (Five Factor Model, regulatory focus, and narcissism) of developers and the introduction and removal of TD. To this end, we complement an existing TD dataset with novel selfreport personality data gathered by surveying developers, and analyze 2,145 commits from 19 developers. Results: We find that conscientiousness, emotional stability, openness to experience, and prevention focus are negatively associated with TD. There were no significant results for extraversion, agreeableness, promotion focus, or narcissism. Conclusions: We take our results as first evidence that developer personality has a systematic influence on the introduction and removal of TD. This has implications not only for future research, which could, for example, study the effects of personality on downstream consequences of TD like defects, but also for software engineering practitioners who may, for example, consider developer personality in staffing decisions.

A Large-Scale, In-Depth Analysis Of Developers' Personalities In The Apache Ecosystem

Abstract: Context Large-scale distributed projects are typically the results of collective efforts performed by multiple developers with heterogeneous personalities. Objective We aim to find evidence that personalities can explain developers’ behavior in large scale-distributed projects. For example, the propensity to trust others — a critical factor for the success of global software engineering — has been found to influence positively the result of code reviews in distributed projects. Method In this paper, we perform a quantitative analysis of ecosystem-level data from the code commits and email messages contributed by the developers working on the Apache Software Foundation (ASF) projects, as representative of large scale-distributed projects. Results We find that there are three common types of personality profiles among Apache developers, characterized in particular by their level of Agreeableness and Neuroticism. We also confirm that developers’ personality is stable over time. Moreover, personality traits do not vary with their role, membership, and extent of contribution to the projects. We also find evidence that more open developers are more likely to make contributors to Apache projects. Conclusion Overall, our findings reinforce the need for future studies on human factors in software engineering to use psychometric tools to control for differences in developers’ personalities.


These figures generally match with those in previous research (Kementchedjhieva and Søgaard 2021), #REFR have discovered.

To investigate how early the warning is triggered for a conversation, we perform an analysis on the two datasets.

Figure 5 shows the distribution of the percentage of utterances that have elapsed when the Hierarchical-Multi model makes a prediction of derailment for all the positive samples in the testing set.

We observe that around 80% of warnings are issued when fewer than five utterances have been seen by the model and the average length of the derailing conversations in the testing set is 7.2.

We also show the average number of utterances between the issuance of a warning and the derailment utterance for four models.

The distance is around 3 for the WTP dataset and 4 for the CMV dataset.

Conversation Modeling To Predict Derailment

Abstract: Conversations among online users sometimes derail, i.e., break down into personal attacks. Such derailment has a negative impact on the healthy growth of cyberspace communities. The ability to predict whether ongoing conversations are likely to derail could provide valuable real-time insight to interlocutors and moderators. Prior approaches predict conversation derailment retrospectively without the ability to forestall the derailment proactively. Some works attempt to make dynamic prediction as the conversation develops, but fail to incorporate multisource information, such as conversation structure and distance to derailment. We propose a hierarchical transformer-based framework that combines utterance-level and conversation-level information to capture fine-grained contextual semantics. We propose a domain-adaptive pretraining objective to integrate conversational structure information and a multitask learning scheme to leverage the distance from each utterance to derailment. An evaluation of our framework on two conversation derailment datasets yields improvement over F1 score for the prediction of derailment. These results demonstrate the effectiveness of incorporating multisource information.

Trouble On The Horizon: Forecasting The Derailment Of Online Conversations As They Develop

Abstract: Online discussions often derail into toxic exchanges between participants. Recent efforts mostly focused on detecting antisocial behavior after the fact, by analyzing single comments in isolation. To provide more timely notice to human moderators, a system needs to preemptively detect that a conversation is heading towards derailment before it actually turns toxic. This means modeling derailment as an emerging property of a conversation rather than as an isolated utterance-level event. ::: Forecasting emerging conversational properties, however, poses several inherent modeling challenges. First, since conversations are dynamic, a forecasting model needs to capture the flow of the discussion, rather than properties of individual comments. Second, real conversations have an unknown horizon: they can end or derail at any time; thus a practical forecasting model needs to assess the risk in an online fashion, as the conversation develops. In this work we introduce a conversational forecasting model that learns an unsupervised representation of conversational dynamics and exploits it to predict future derailment as the conversation develops. By applying this model to two new diverse datasets of online conversations with labels for antisocial events, we show that it outperforms state-of-the-art systems at forecasting derailment.


B. Squeeze-And-Excitation Networks Attention Mechanism

The SE attention mechanism encodes the entire spatial feature in a channel as a global feature through automatic learning utilizing global averaging pooling by means of the squeeze operation #REFR .

where z c is the output associated with the c channel and the input x is a convolutional layer with a fixed kernel size.

An excitation operation is then performed to obtain the importance of each channel in the feature map.

σ is the sigmoid activation function and δ is the relu activation function.

Learning Behavior Recognition In Smart Classroom With Multiple Students Based On Yolov5

Abstract: Deep learning-based computer vision technology has grown stronger in recent years, and cross-fertilization using computer vision technology has been a popular direction in recent years. The use of computer vision technology to identify students' learning behavior in the classroom can reduce the workload of traditional teachers in supervising students in the classroom, and ensure greater accuracy and comprehensiveness. However, existing student learning behavior detection systems are unable to track and detect multiple targets precisely, and the accuracy of learning behavior recognition is not high enough to meet the existing needs for the accurate recognition of student behavior in the classroom. To solve this problem, we propose a YOLOv5s network structure based on you only look once (YOLO) algorithm to recognize and analyze students' classroom behavior in this paper. Firstly, the input images taken in the smart classroom are pre-processed. Then, the pre-processed image is fed into the designed YOLOv5 networks to extract deep features through convolutional layers, and the Squeeze-and-Excitation (SE) attention detection mechanism is applied to reduce the weight of background information in the recognition process. Finally, the extracted features are classified by the Feature Pyramid Networks (FPN) and Path Aggregation Network (PAN) structures. Multiple groups of experiments were performed to compare with traditional learning behavior recognition methods to validate the effectiveness of the proposed method. When compared with YOLOv4, the proposed method is able to improve the mAP performance by 11%.

Squeeze-And-Excitation Networks

Abstract: Convolutional neural networks are built upon the convolution operation, which extracts informative features by fusing spatial and channel-wise information together within local receptive fields. In order to boost the representational power of a network, much existing work has shown the benefits of enhancing spatial encoding. In this work, we focus on channels and propose a novel architectural unit, which we term the"Squeeze-and-Excitation"(SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channels. We demonstrate that by stacking these blocks together, we can construct SENet architectures that generalise extremely well across challenging datasets. Crucially, we find that SE blocks produce significant performance improvements for existing state-of-the-art deep architectures at slight computational cost. SENets formed the foundation of our ILSVRC 2017 classification submission which won first place and significantly reduced the top-5 error to 2.251%, achieving a 25% relative improvement over the winning entry of 2016.

automatic learning
entire spatial feature


YOLOv4 achieves superior performance with these improvements and modifications.YOLOv5 adds data enhancement and FOCUs modules to the v3 backbone and replaces IOU with GIOU #REFR and NMS with DiouNMS respectively.

Consequently, the one-stage detectors have more efficient detection and require less computer performance.

OverFeat #OTHEREFR was the first application of a CNN-based single-stage target detector.

YOLO #OTHEREFR and SSD #OTHEREFR were introduced after it, and SSD achieved great performance.YOLO and YOLOv2 #OTHEREFR only use the last output feature of the backbone.

They can both achieve a very fast detection speed, but at the expense of accuracy.

YOLOv3 #OTHEREFR is based on YoLov1, with an FPN architecture and a backbone that was changed from DarkNet-19 to DarkNet-53.

Our improved network structure will be compared with YOLOv5 with backbone network of CSPDarkNet53 #OTHEREFR .

The FPN works as the basis for a detector model that is also used in a variety of popular models, such as RetinaNet #OTHEREFR , FCOS #OTHEREFR , and their variants.

A Novel Multi To Single Module For Small Object Detection

Abstract: Small object detection presents a significant challenge in computer vision and object detection. The performance of small object detectors is often compromised by a lack of pixels and less significant features. This issue stems from information misalignment caused by variations in feature scale and information loss during feature processing. In response to this challenge, this paper proposes a novel the Multi to Single Module (M2S), which enhances a specific layer through improving feature extraction and refining features. Specifically, M2S includes the proposed Cross-scale Aggregation Module (CAM) and explored Dual Relationship Module (DRM) to improve information extraction capabilities and feature refinement effects. Moreover, this paper enhances the accuracy of small object detection by utilizing M2S to generate an additional detection head. The effectiveness of the proposed method is evaluated on two datasets, VisDrone2021-DET and SeaDronesSeeV2. The experimental results demonstrate its improved performance compared with existing methods. Compared to the baseline model (YOLOv5s), M2S improves the accuracy by about 1.1% on the VisDrone2021-DET testing dataset and 15.68% on the SeaDronesSeeV2 validation set. 1 Introduction Small object detection ([1],[2],[3],[4],[5]) has long been a challenge in object detection, which aims to accurately detect small objects (those under 32 pixels by 32 pixels) with very few visual features in the image. A detection model consisting of a backbone network, an encoder, and a decoder is proposed in You Only Look One-level Feature (YOLOF) [6]. It is proposed in YOLOF that by choosing the right scale features for output to a particular level with multiple-input and single-output, comparable performance to multiple-input and multiple-output can be obtained. In the small object detection task, this paper proposes a multiple-input to the single-output module that outputs with a low-level feature with high resolution. It provides the detector with more accurate and richer information, thereby improving its performance. Feature Pyramid network (FPN) [7] success is due to the idea of divide and conquer. Divide and conquer ensures that image features are available in different layers, whereas if only a unidirectional flow of information is delivered, then FPN ends up with a limitation at each layer. The result is that each layer of character can only be thinking within its own perspective. Thus, previous work has been devoted to increasing the interaction of different levels of feature and allowing more global thinking at each level. FPN appears as the backbone network of excellent detectors SSD [8], YOLO [9], and RCNN [10]. EffientDet [11]uses a cross-edge connecting BiFPN and repeats it to obtain better accuracy and efficiency. Although the previous works have shown gratifying performance, object detection still has a problem: the poor detection of small-scale instance objects. Based on this problem, this paper proposes a module to enhance communication at different network levels and improve feature extraction efficiency. The Cross-scale Fusion Module

Generalized Intersection Over Union: A Metric And A Loss For Bounding Box Regression

Abstract: Intersection over Union (IoU) is the most popular evaluation metric used in the object detection benchmarks. However, there is a gap between optimizing the commonly used distance losses for regressing the parameters of a bounding box and maximizing this metric value. The optimal objective for a metric is the metric itself. In the case of axis-aligned 2D bounding boxes, it can be shown that IoU can be directly used as a regression loss. However, IoU has a plateau making it infeasible to optimize in the case of non-overlapping bounding boxes. In this paper, we address the this weakness by introducing a generalized version of IoU as both a new loss and a new metric. By incorporating this generalized IoU ( GIoU) as a loss into the state-of-the art object detection frameworks, we show a consistent improvement on their performance using both the standard, IoU based, and new, GIoU based, performance measures on popular object detection benchmarks such as PASCAL VOC and MS COCO.


In addition, we are not tied to a particular annotation from datasets (e.g., specific attributes in CelebA #REFR ).

Consider a user interested in evaluating which factors contribute to the lack of robustness in a cat/dog classifier (target model).

By selecting a list of keyword attributes, the user is able to (1) see counterfactual images where semantic variations flip the target model predictions (see the classifier score in the top-right corner of the counterfactual images) and #OTHEREFR quantify the sensitivity of each attribute for the target model (see sensitivity histogram on the top).

Instead of using a test set, we propose using a StyleGAN generator as the picture engine for sampling counterfactual images.

CLIP transforms user's text input, and enables model diagnosis in an open-vocabulary setting.

This is a major advantage since there is no need for collecting and annotating images and minimal user expert knowledge.

To summarize, our proposed work offers three major improvements over earlier efforts:

• The user requires neither a labeled, balanced test dataset, and minimal expert knowledge in order to evaluate where a model fails (i.e., model diagnosis).

In addition, the method provides a sensitivity histogram across the attributes of interest.

• When a different target model or a new user-defined attribute space is introduced, it is not necessary to retrain our system, allowing for practical use.

• The target model fine-tuned with counterfactual images not only slightly improves the classification performance, but also greatly increases the distributional robustness against counterfactual images.

Zero-Shot Model Diagnosis

Abstract: When it comes to deploying deep vision models, the behavior of these systems must be explicable to ensure confidence in their reliability and fairness. A common approach to evaluate deep learning models is to build a labeled test set with attributes of interest and assess how well it performs. However, creating a balanced test set (i.e., one that is uniformly sampled over all the important traits) is often time-consuming, expensive, and prone to mistakes. The question we try to address is: can we evaluate the sensitivity of deep learning models to arbitrary visual attributes without an annotated test set? This paper argues the case that Zero-shot Model Diagnosis (ZOOM) is possible without the need for a test set nor labeling. To avoid the need for test sets, our system relies on a generative model and CLIP. The key idea is enabling the user to select a set of prompts (relevant to the problem) and our system will automatically search for semantic counterfactual images (i.e., synthesized images that flip the prediction in the case of a binary classifier) using the generative model. We evaluate several visual tasks (classification, key-point detection, and segmentation) in multiple visual domains to demonstrate the viability of our methodology. Extensive experiments demonstrate that our method is capable of producing counterfactual images and offering sensitivity analysis for model diagnosis without the need for a test set. * Equal contribution. † The code is publicly available at the project page: https://zero-shotmodel-diagnosis.github.io/. CLIP StyleGAN Generator How would a change in [attribute] affect [my model]'s prediction? User Diagnosis Request Neural Tools Diagnosis Outcomes [attribute] = "green eye" "vertical pupil" "pointed ear" ... [my model] = cat/dog classifier ... Zero-shot counterfactual images Zero-shot sensitivity histogram "g re en ey e" "v er ti ca l p u p il " "p o in te d ea r" ... ...

Deep Learning Face Attributes In The Wild

Abstract: Predicting face attributes in the wild is challenging due to complex face variations. We propose a novel deep learning framework for attribute prediction in the wild. It cascades two CNNs, LNet and ANet, which are fine-tuned jointly with attribute tags, but pre-trained differently. LNet is pre-trained by massive general object categories for face localization, while ANet is pre-trained by massive face identities for attribute prediction. This framework not only outperforms the state-of-the-art with a large margin, but also reveals valuable facts on learning face representation. (1) It shows how the performances of face localization (LNet) and attribute prediction (ANet) can be improved by different pre-training strategies. (2) It reveals that although the filters of LNet are fine-tuned only with image-level attribute tags, their response maps over entire images have strong indication of face locations. This fact enables training LNet for face localization with only image-level annotations, but without face bounding boxes or landmarks, which are required by all attribute recognition works. (3) It also demonstrates that the high-level hidden neurons of ANet automatically discover semantic concepts after pre-training with massive face identities, and such concepts are significantly enriched after fine-tuning with attribute tags. Each attribute can be well explained with a sparse linear combination of these concepts.


G.4. Extensibility To Different Line Detectors And Matchers

While endpoint-based line matcher achieves very promising performance in Table 13 , the endpoints of the matched line still do not necessarily match each other, indicating that the performance gain may come from the effectiveness of advanced point features #REFR .

The endpoint-based line matcher is surprisingly effective, Table 13 .

Extensibility of the framework to different line detectors and matchers.

We show results of "R1 / P1 / #supports" for line mapping on Hypersim #OTHEREFR with only line-line proposals.

Also, we present here two customized line matchers by matching the line endpoints with either the nearest neighbor strategy ("Endpts SP + NN") or an advanced point-based matcher SuperGlue #OTHEREFR ("Endpts SP + SG"). Table 14 .

Comparison between endpoint and line triangulation with "Endpts SP + SG" matcher on Hypersim #OTHEREFR .

as "Endpoint SP + SG" consistently achieves the best track association under all detectors.

We further test again the comparison between endpoint triangulation and algebraic line triangulation to see whether the endpoints of the matched line from the endpoint-based line matcher correspond to each other.

Results in Table 14 show similar trends as Table  3 in the main paper, where performing algebraic line triangulation is significantly better than directly triangulating endpoints.

This finding indicates that the endpoint-based matcher is surprisingly effective on matching lines with, however, unmatched endpoints, which may be due to the advantages of the rich point features #OTHEREFR .

This encourages more research towards integrating the success of existing point description and matching solutions to improve line matching.

3D Line Mapping Revisited

Abstract: In contrast to sparse keypoints, a handful of line segments can concisely encode the high-level scene layout, as they often delineate the main structural elements. In addition to offering strong geometric cues, they are also omnipresent in urban landscapes and indoor scenes. Despite their apparent advantages, current line-based reconstruction methods are far behind their point-based counterparts. In this paper we aim to close the gap by introducing LIMAP, a library for 3D line mapping that robustly and efficiently creates 3D line maps from multi-view imagery. This is achieved through revisiting the degeneracy problem of line triangulation, carefully crafted scoring and track building, and exploiting structural priors such as line coincidence, parallelism, and orthogonality. Our code integrates seamlessly with existing point-based Structure-from-Motion methods and can leverage their 3D points to further improve the line reconstruction. Furthermore, as a byproduct, the method is able to recover 3D association graphs between lines and points / vanishing points (VPs). In thorough experiments, we show that LIMAP significantly outperforms existing approaches for 3D line mapping. Our robust 3D line maps also open up new research directions. We show two example applications: visual localization and bundle adjustment, where integrating lines alongside points yields the best results. Code is available at https://github.com/cvg/limap.

Superpoint: Self-Supervised Interest Point Detection And Description

Abstract: This paper presents a self-supervised framework for training interest point detectors and descriptors suitable for a large number of multiple-view geometry problems in computer vision. As opposed to patch-based neural networks, our fully-convolutional model operates on full-sized images and jointly computes pixel-level interest point locations and associated descriptors in one forward pass. We introduce Homographic Adaptation, a multi-scale, multi-homography approach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g., synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able to repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other traditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches when compared to LIFT, SIFT and ORB.


Due to the complex nature of the problem, we solve it via exhaustive search, performing line search over the integer values of τ in the range given in #REFR and obtain the optimum of α ∈ (0, 1] corresponding to each value of τ .

Formulation P is a non-convex mixed-integer programming problem.

Note that, given a value of τ , P is still non-convex with respect to α.

Therefore, we discretize the search space of α and perform a line search over the discretized search space of α for each τ .

Based on this approach, the search space of P remains to be small due to the limited ranges/choices of τ and α (i.e., the time complexity of performing line search over τ is O(T − t k − ∆ k−1 ) and the time complexity of performing line search over α is O(1/S α ), where S α is the discretization step used to discretize (0, 1] interval, resulting in a time complexity of O((T − t k − ∆ k−1 ) × 1/S α )).

Thus, we are able to solve the problem via a reasonable precision (e.g., S α = 0.01) in a short duration of time (e.g., less than 3 seconds on a laptop with Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz).

Delay-Aware Hierarchical Federated Learning

Abstract: Federated learning has gained popularity as a means of training models distributed across the wireless edge. The paper introduces delay-aware federated learning (DFL) to improve the efficiency of distributed machine learning (ML) model training by addressing communication delays between edge and cloud. DFL employs multiple stochastic gradient descent iterations on device datasets during each global aggregation interval and intermittently aggregates model parameters through edge servers in local subnetworks. The cloud server synchronizes the local models with the global deployed model computed via a local-global combiner at global synchronization. The convergence behavior of DFL is theoretically investigated under a generalized data heterogeneity metric. A set of conditions is obtained to achieve the sub-linear convergence rate of O(1/k). Based on these findings, an adaptive control algorithm is developed for DFL, implementing policies to mitigate energy consumption and edge-to-cloud communication latency while aiming for a sublinear convergence rate. Numerical evaluations show DFL's superior performance in terms of faster global model convergence, reduced resource consumption, and robustness against communication delays compared to existing FL algorithms. In summary, this proposed method offers improved efficiency and satisfactory results when dealing with both convex and non-convex loss functions.

On The Convergence Of Fedavg On Non-Iid Data

Abstract: Federated learning enables a large amount of edge computing devices to jointly learn a model without data sharing. As a leading algorithm in this setting, Federated Averaging (\texttt{FedAvg}) runs Stochastic Gradient Descent (SGD) in parallel on a small subset of the total devices and averages the sequences only once in a while. Despite its simplicity, it lacks theoretical guarantees under realistic settings. In this paper, we analyze the convergence of \texttt{FedAvg} on non-iid data and establish a convergence rate of $\mathcal{O}(\frac{1}{T})$ for strongly convex and smooth problems, where $T$ is the number of SGDs. Importantly, our bound demonstrates a trade-off between communication-efficiency and convergence rate. As user devices may be disconnected from the server, we relax the assumption of full device participation to partial device participation and study different averaging schemes; low device participation rate can be achieved without severely slowing down the learning. Our results indicate that heterogeneity of data slows down the convergence, which matches empirical observations. Furthermore, we provide a necessary condition for \texttt{FedAvg} on non-iid data: the learning rate $\eta$ must decay, even if full-gradient is used; otherwise, the solution will be $\Omega (\eta)$ away from the optimal.


However,developing reinforcement learning solutions to increase the safety and robustness of learned strategies in healthcare remains a key challenge #REFR .

While this method is computationally tractable for small POMDPs, the resulting belief MDP is often too large to optimize effectively (Ç elik et al., 2015) .

Thus in many large scale setting approximate dynamic programming techniques have been proposed to approximate optimal policies #OTHEREFR .

Our setting differs from the POMDP setting in that the transition function is also partially observable since each patient has varying metabolism rate that makes the belief state conversion challenging to implement.

Both reinforcement learning #OTHEREFR and POMDPs have been applied in the setting of personalized healthcare #OTHEREFR .

Several reinforcement learning methods have been proposed for personalized medicine including deep reinforcement learning #OTHEREFR , off policy learning #OTHEREFR , and some model based methods #OTHEREFR .

POMDPs have also been used to study personalized dosing in applications such as heart disease #OTHEREFR , stroke #OTHEREFR ) and

POMDP models have also been used in population health and disease screening and disease management #OTHEREFR . #OTHEREFR

(2021) incorporated a personalized readmission prediction model to manage patient flow in a hospital.

The methods we propose in this paper expand on these methods to a healthcare setting where both states and dynamics are partially observed.

Model Based Reinforcement Learning For Personalized Heparin Dosing

Abstract: One of the key challenges in sequential decision making is optimizing systems safely in the case of partial information. While much of the existing work has focused on addressing this challenge in the case of either partially known states or partially known system dynamics, it is further exacerbated in cases where both states and dynamics are partially known. For instance the setting of computing heparin doses for patients fits this paradigm since the concentration of heparin in the patient cannot be measured directly and the rates at which patients metabolize it vary greatly between individuals. While many approaches proposed to resolve the challenge in this setting are model free, they require complex models that are not transparent to decision makers, and are difficult to analyze and guarantee safety. However, if some of the structure of the dynamics is known, a model based approach can be leveraged to provide safe policies with practical empirical performance and theoretical worst case guarantees. In this paper we propose a model based framework to address the challenge of partially observed states and dynamics in the context of designing personalized doses of heparin. We use a predictive model based on pharmacokinetics (the study of how the body effects substances through absorption, distribution, and metabolism) parameterized individually by patient, and infer the current concentration of heparin and predict future therapeutic effects taking into account different patients' characteristics. We formulate the patient parameter estimation problem in to a mixed integer linear program and show that our estimates are statistically consistent. We leverage this model by developing an adaptive dosing algorithm that outputs asymptotically optimal dose sequences based on a scenario generation approach, this approach is also capable of ensuring that the required heparin doses are maintained within a safe level. We validate our models with numerical experiments by first comparing the predictive capabilities of our model against existing machine learning techniques and demonstrating how our dosing algorithm can keep patients' related medical tests within a therapeutic range in a simulated ICU environment. Our results show that our methods are capable of maintaining patients in therapeutic range for 87.7% of the treatment time as opposed to existing weight based protocols that can only do so for 55.6% of the treatment time.

Reinforcement Learning In Healthcare: A Survey

Abstract: As a subfield of machine learning, \emph{reinforcement learning} (RL) aims at empowering one's capabilities in behavioural decision making by using interaction experience with the world and an evaluative feedback. Unlike traditional supervised learning methods that usually rely on one-shot, exhaustive and supervised reward signals, RL tackles with sequential decision making problems with sampled, evaluative and delayed feedback simultaneously. Such distinctive features make RL technique a suitable candidate for developing powerful solutions in a variety of healthcare domains, where diagnosing decisions or treatment regimes are usually characterized by a prolonged and sequential procedure. This survey will discuss the broad applications of RL techniques in healthcare domains, in order to provide the research community with systematic understanding of theoretical foundations, enabling methods and techniques, existing challenges, and new insights of this emerging paradigm. By first briefly examining theoretical foundations and key techniques in RL research from efficient and representational directions, we then provide an overview of RL applications in a variety of healthcare domains, ranging from dynamic treatment regimes in chronic diseases and critical care, automated medical diagnosis from both unstructured and structured clinical data, as well as many other control or scheduling domains that have infiltrated many aspects of a healthcare system. Finally, we summarize the challenges and open issues in current research, and point out some potential solutions and directions for future research.


These complexity upper bounds coincide with the ones provided in #REFR in the case of safety automata.

In this paper we ultimately solve Problem 1.1 for non-deterministic parity tree automata, which capture the whole MSO. Theorem 1.2.

There is an algorithm that inputs a non-deterministic parity tree automaton A and computes the coin-flipping measure of the language of trees recognised by A. Moreover, this measure is an algebraic number.

As a consequence, given additionally a rational number q, we can decide if the resulting measure is equal, smaller, or greater than q.

Our algorithm is able to produce a representation of the above measure in three-fold exponential time.

The decision problems with a fixed rational number q can be solved in two-fold exponential space.

It is worth to note that natural stronger probabilistic variants of the satisfiability problem are known to be undecidable.

For example, both the non-emptiness problem #OTHEREFR and the value-1 problem #OTHEREFR for probabilistic finite automata, even for finite words, are undecidable.

Moreover, Baier, Bertrand, and Grösser #OTHEREFR showed that it is undecidable whether there exists an ω-word that is accepted by a given probabilistic Büchi automaton with a positive probability.

So presumably we are close to the frontiers of decidability here.

One should note that our result does not entail the Rabin tree theorem-the satisfiability problem does not directly reduce to its probabilistic variant.

The Probabilistic Rabin Tree Theorem *

Abstract: The Rabin tree theorem yields an algorithm to solve the satisfiability problem for monadic second-order logic over infinite trees. Here we solve the probabilistic variant of this problem. Namely, we show how to compute the probability that a randomly chosen tree satisfies a given formula. We additionally show that this probability is an algebraic number. This closes a line of research where similar results were shown for formalisms weaker than the full monadic second-order logic.

The Uniform Measure Of Simple Regular Sets Of Infinite Trees

Abstract: We consider the problem of computing the measure of a regular set of infinite binary trees. While the general case remains unsolved, we show that the measure of a language can be computed when the set is given in one of the following three formalisms: a first-order formula with no descendant relation; a Boolean combination of conjunctive queries (with descendant relation); or by a non-deterministic safety tree automaton. Additionally, in the first two cases the measure of the set is always rational, while in the third it is an algebraic number. Moreover, we provide an example of a first-order formula that uses descendant relation and defines a language of infinite trees having an irrational (but algebraic) measure.


These algorithms demonstrate a giant leap in 3D detection performance on multi-camera datasets (E.g., Nuscenes #REFR ).

Our proposed VEDet encodes the 3D scene from multiview images, and decodes objects with view-conditioned queries.

The predicted 3D bounding boxes are expressed in the underlying views of the queries, which enables us to enforce viewpoint equivariance among predictions from multiple views.

Virtual query views are generated during training and together with the viewpoint equivariance regularization bring richer geometric learning signals to guide the model to better understand the 3D structure in the scene.

During inference, the global predictions can be obtained by simply choosing the global frame as the query view.

multi-view camera-based 3D algorithms #OTHEREFR proposed to jointly aggregate multi-view information at the feature level, and directly predict a single set of detections in the global frame.

To aggregate information from different views, one line of query-based detectors adopt transformers to query image features #OTHEREFR or bird's-eye-view (BEV) features #OTHEREFR via an attention mechanism.

In contrast, another line of works "lift-splat-shoot" #OTHEREFR image features from each view into the shared BEV features to be processed by convolutional detection heads #OTHEREFR .

To further mitigate the depth ambiguity, some concurrent works have started extending multi-view to "multi-sweep" across timestamps and observe a promising performance boost #OTHEREFR .

While the works mentioned above demonstrate a strong potential for multi-view 3D detection, progress has concentrated on input aggregation and information interplay across frames and less on learning objectives.

We argue that the learning objective can play a crucial role in ingesting the core knowledge in a multi-view setting: 3D geometry.

Viewpoint Equivariance For Multi-View 3D Object Detection

Abstract: 3D object detection from visual sensors is a cornerstone capability of robotic systems. State-of-the-art methods focus on reasoning and decoding object bounding boxes from multi-view camera input. In this work we gain intuition from the integral role of multi-view consistency in 3D scene understanding and geometric learning. To this end, we introduce VEDet, a novel 3D object detection framework that exploits 3D multi-view geometry to improve localization through viewpoint awareness and equivariance. VEDet leverages a query-based transformer architecture and encodes the 3D scene by augmenting image features with positional encodings from their 3D perspective geometry. We design view-conditioned queries at the output level, which enables the generation of multiple virtual frames during training to learn viewpoint equivariance by enforcing multi-view consistency. The multi-view geometry injected at the input level as positional encodings and regularized at the loss level provides rich geometric cues for 3D object detection, leading to state-of-the-art performance on the nuScenes benchmark. The code and model are made available at https://github.com/TRI-ML/VEDet.

Nuscenes: A Multimodal Dataset For Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image-based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first published dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online at this http URL.

3D detection performance
multi-camera datasets


This method regards the TRADES #REFR , a state-of-the-art adversarial training method, as a special case with only zero-th term, and is able to locate flatter minima.

It was developed to guarantee weight smoothness by simply averaging various checkpoints throughout the training trajectory.

SWA has been applied to semi-supervised learning, Bayesian inference, and low-precision training.

Especially, #OTHEREFR used SWA for the first time in adversarial training in order to smooth the weights and locate flatter minima, and showed that it enhances the adversarial generalization.

Different from SWA, we propose a novel method (with details provided in Sec.

4) via Taylor expansion of a small Gaussian noise to smooth the update of weights.

To demonstrate the latter, we visualize the loss landscape with respect to both input and weight spaces. As shown in Fig.

2 , compared to the TRADES, our method significantly flattens the rugged landscape.

In addition, it is worth noting that our empirical results (in Sec.

5) demonstrate that our method can further improve not only clean accuracy but also adversarial accuracy compared with other SOTA (includes SWA from #OTHEREFR ).

Randomized Adversarial Training Via Taylor Expansion

Abstract: In recent years, there has been an explosion of research into developing more robust deep neural networks against adversarial examples. Adversarial training appears as one of the most successful methods. To deal with both the robustness against adversarial examples and the accuracy over clean examples, many works develop enhanced adversarial training methods to achieve various trade-offs between them [19, 38, 82]. Leveraging over the studies [8, 32] that smoothed update on weights during training may help find flat minima and improve generalization, we suggest reconciling the robustness-accuracy trade-off from another perspective, i.e., by adding random noise into deterministic weights. The randomized weights enable our design of a novel adversarial training method via Taylor expansion of a small Gaussian noise, and we show that the new adversarial training method can flatten loss landscape and find flat minima. With PGD, CW, and Auto Attacks, an extensive set of experiments demonstrate that our method enhances the state-of-the-art adversarial training methods, boosting both robustness and clean accuracy. The code is available at https://github.com/Alexkael/Randomized-Adversarial-Training.

Theoretically Principled Trade-Off Between Robustness And Accuracy

Abstract: We identify a trade-off between robustness and accuracy that serves as a guiding principle in the design of defenses against adversarial examples. Although this problem has been widely studied empirically, much remains unknown concerning the theory underlying this trade-off. In this work, we decompose the prediction error for adversarial examples (robust error) as the sum of the natural (classification) error and boundary error, and provide a differentiable upper bound using the theory of classification-calibrated loss, which is shown to be the tightest possible upper bound uniform over all probability distributions and measurable predictors. Inspired by our theoretical analysis, we also design a new defense method, TRADES, to trade adversarial robustness off against accuracy. Our proposed algorithm performs well experimentally in real-world datasets. The methodology is the foundation of our entry to the NeurIPS 2018 Adversarial Vision Challenge in which we won the 1st place out of ~2,000 submissions, surpassing the runner-up approach by $11.41\%$ in terms of mean $\ell_2$ perturbation distance.


To alleviate this obstacle, #REFR propose to utilize natural color distributions learned from large-scale image data when no appropriate matching region is available in the reference.

With the advent of deep learning, #OTHEREFR proposed a real-time user interactive colorization model which can generate visually plausible results with sparse color hints.

However, these methods may fail to provide robust and automatic colorization when dense user annotations are not available.

Reference-based Colorization methods colorize the target grayscale image by transferring color information from a reference image using global color #OTHEREFR or some local similarity #OTHEREFR .

Recently, some researchers #OTHEREFR have made use of deep features extracted by a pretrained neural network for reliable spatial correspondence and generate plausible results.

However, these methods usually require a content-related image as reference, which poses challenges to users.

Automatic Colorization is generally a kind of learningbased method, which uses neural networks to learn the mapping of grayscale images to color images #OTHEREFR .

These methods can be trained on large-scale datasets and do not require user input or reference images.

Some follow-up works promote the colorized results by integrating additional semantic guidance, like class labels #OTHEREFR , semantic segmentation maps #OTHEREFR and instance-level bounding box #OTHEREFR .

To improve the diversity of the colors generated, #OTHEREFR proposed a method that uses a convolutional neural network (CNN) to predict a probability distribution over the possible colors for each pixel.

Generative Adversarial Network (GAN) #OTHEREFR is adopted for realistic and vivid colorization #OTHEREFR .

Improved Diffusion-Based Image Colorization Via Piggybacked Models

Abstract: Image colorization has been attracting the research interests of the community for decades. However, existing methods still struggle to provide satisfactory colorized results given grayscale images due to a lack of human-like global understanding of colors. Recently, large-scale Textto-Image (T2I) models have been exploited to transfer the semantic information from the text prompts to the image domain, where text provides a global control for semantic objects in the image. In this work, we introduce a colorization model piggybacking on the existing powerful T2I diffusion model. Our key idea is to exploit the color prior knowledge in the pre-trained T2I diffusion model for realistic and diverse colorization. A diffusion guider is designed to incorporate the pre-trained weights of the latent diffusion model to output a latent color prior that conforms to the visual semantics of the grayscale input. A lightness-aware VQVAE will then generate the colorized result with pixelperfect alignment to the given grayscale image. Our model can also achieve conditional colorization with additional inputs (e.g. user hints and texts). Extensive experiments show that our method achieves state-of-the-art performance in terms of perceptual quality.

Deep Exemplar-Based Colorization

Abstract: We propose the first deep learning approach for exemplar-based local colorization. Given a reference color image, our convolutional neural network directly maps a grayscale image to an output colorized image. Rather than using hand-crafted rules as in traditional exemplar-based methods, our end-to-end colorization network learns how to select, propagate, and predict colors from the large-scale data. The approach performs robustly and generalizes well even when using reference images that are unrelated to the input grayscale image. More importantly, as opposed to other learning-based colorization methods, our network allows the user to achieve customizable results by simply feeding different references. In order to further reduce manual effort in selecting the references, the system automatically recommends references with our proposed image retrieval algorithm, which considers both semantic and luminance information. The colorization can be performed fully automatically by simply picking the top reference suggestion. Our approach is validated through a user study and favorable quantitative comparisons to the-state-of-the-art methods. Furthermore, our approach can be naturally extended to video colorization. Our code and models will be freely available for public use.


A.1. Diswot Under Different Teacher Models.

According to the results in Table 13, the accuracy of the student network ResNet20 #REFR is unable to make consistent gains as the size of the teacher network increases.

When the teacher model becomes larger, the fixed handdesigned model would have huge teacher-student gaps, limiting the performance gain.

DisWOT aims to solve this problem by searching the suitable student architecture for different teacher models.

Our proposed DisWOT enables a consistent increase in student network performance as the teacher network capacity increases on search space S 0 .

In addition, DisWOT † achieves a performance gain of about 2% when stronger distillers are adopted. Table 13 .

Top-1 accuracy (%) of ResNet20 with KD #OTHEREFR , student (DisWOT) with KD #OTHEREFR , student (DisWOT) with DisWOT † on search space S0 under different teachers.

Diswot: Student Architecture Search For Distillation Without Training

Abstract: Knowledge distillation (KD) is an effective training strategy to improve the lightweight student models under the guidance of cumbersome teachers. However, the large architecture difference across the teacher-student pairs limits the distillation gains. In contrast to previous adaptive distillation methods to reduce the teacher-student gap, we explore a novel training-free framework to search for the best student architectures for a given teacher. Our work first empirically show that the optimal model under vanilla training cannot be the winner in distillation. Secondly, we find that the similarity of feature semantics and sample relations between random-initialized teacher-student networks have good correlations with final distillation performances. Thus, we efficiently measure similarity matrixs conditioned on the semantic activation maps to select the optimal student via an evolutionary algorithm without any training. In this way, our student architecture search for Distillation WithOut Training (DisWOT) significantly improves the performance of the model in the distillation stage with at least 180× training acceleration. Additionally, we extend similarity metrics in DisWOT as new distillers and KD-based zero-proxies. Our experiments on CIFAR, ImageNet and NAS-Bench-201 demonstrate that our technique achieves state-of-the-art results on different search spaces. Our project and code are available at https://lilujunai.github.io/DisWOT-CVPR2023/. * Corresponding author, † equal contribution, PD conducted main experiments, LL proposed ideas and led the project & writing.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


Large-Batch Training Challenge And Improvement

This is aligned with the observations in ERM #REFR , where the lack of stochasticity makes the optimizer difficult to escape from a sharp local minimum.

Recall that the full batch size (50k) was used in the existing IRM implementations (Arjovsky et al., 2019; #OTHEREFR ).

As we can see, in the full-batch setup, IRM methods lead to widely different invariant prediction accuracies, where REX and IRMV1 significantly outperform IRMV0.

In contrast, in the small-batch case (with size 1k), the discrepancy in accuracy across methods vanishes.

We see that IRMv0 can be as effective as IRMv1 and other IRM variants (such as REX) only if an appropriate small batch size is used.

Empirical evidence in Figure 1 shows that large-batch IRM training is less effective than smallbatch.

We also justify this issue by visualizing the loss landscapes in Figure A2 .

Notably, the small-batch training enables IRMV1 to converge to a local optimum with a flat loss landscape, indicating better generalization #OTHEREFR .

Small-batch training is effective versus a zoo of large-batch optimization enhancements.

To mitigate the large-batch IRM training issue, we next investigate the effectiveness of both smallbatch training and a zoo of large-batch optimization enhancements.

Inspired by large-batch training techniques to scale up ERM, we consider Large-batch SGD (LSGD) #OTHEREFR and Layerwise Adaptive Learning Rate (LALR) #OTHEREFR .

What Is Missing In Irm Training And Evalua-Tion? Challenges And Solutions

Abstract: Invariant risk minimization (IRM) has received increasing attention as a way to acquire environment-agnostic data representations and predictions, and as a principled solution for preventing spurious correlations from being learned and for improving models' out-of-distribution generalization. Yet, recent works have found that the optimality of the originally-proposed IRM optimization (IRMV1) may be compromised in practice or could be impossible to achieve in some scenarios. Therefore, a series of advanced IRM algorithms have been developed that show practical improvement over IRMV1. In this work, we revisit these recent IRM advancements, and identify and resolve three practical limitations in IRM training and evaluation. First, we find that the effect of batch size during training has been chronically overlooked in previous studies, leaving room for further improvement. We propose small-batch training and highlight the improvements over a set of large-batch optimization techniques. Second, we find that improper selection of evaluation environments could give a false sense of invariance for IRM. To alleviate this effect, we leverage diversified test-time environments to precisely characterize the invariance of IRM when applied in practice. Third, we revisit Ahuja et al. (2020)'s proposal to convert IRM into an ensemble game and identify a limitation when a single invariant predictor is desired instead of an ensemble of individual predictors. We propose a new IRM variant to address this limitation based on a novel viewpoint of ensemble IRM games as consensus-constrained bilevel optimization. Lastly, we conduct extensive experiments (covering 7 existing IRM variants and 7 datasets) to justify the practical significance of revisiting IRM training and evaluation in a principled manner.

Scaling Sgd Batch Size To 32K For Imagenet Training.

Abstract: The most natural way to speed-up the training of large networks is to use data-parallelism on multiple GPUs. To scale Stochastic Gradient (SG) based methods to more processors, one need to increase the batch size to make full use of the computational power of each GPU. However, keeping the accuracy of network with increase of batch size is not trivial. Currently, the state-of-the art method is to increase Learning Rate (LR) proportional to the batch size, and use special learning rate with "warm-up" policy to overcome initial optimization difficulty. ::: By controlling the LR during the training process, one can efficiently use large-batch in ImageNet training. For example, Batch-1024 for AlexNet and Batch-8192 for ResNet-50 are successful applications. However, for ImageNet-1k training, state-of-the-art AlexNet only scales the batch size to 1024 and ResNet50 only scales it to 8192. The reason is that we can not scale the learning rate to a large value. To enable large-batch training to general networks or datasets, we propose Layer-wise Adaptive Rate Scaling (LARS). LARS LR uses different LRs for different layers based on the norm of the weights and the norm of the gradients. By using LARS algoirithm, we can scale the batch size to 32768 for ResNet50 and 8192 for AlexNet. Large batch can make full use of the system's computational power. For example, batch-4096 can achieve 3x speedup over batch-512 for ImageNet training by AlexNet model on a DGX-1 station (8 P100 GPUs).


Compared with raw dMRI data, the networks trained with fODF peaks data achieve better tract segmentation results #REFR .

We use the dataset from TractSeg #OTHEREFR , including 105 subjects from the Human Connectome Project (HCP) #OTHEREFR . Each sub- Fig. 1 .

Overview of registration-and uncertainty-based framework for WM tract segmentation with only one labeled subject. ject has 72 tracts annotated by neuroanatomists.

We follow the same dataset split as #OTHEREFR , using 84 subjects for training, and the rest 21 subjects for testing.

Regarding our subjectlevel one-shot segmentation setting, we only use one annotated subject out of the 84 training subjects.

As shown in Fig.1 , we use multi-shell multi-tissue constrained spherical deconvolution (CSD) method #OTHEREFR with all gradient directions to transform dMRI data to fiber orientation distribution function (fODF) peaks data, which has 9 channels corresponding to sagittal, axial, and coronal directions (each direction has three channels), as in #OTHEREFR .

Therefore, we directly segment tracts in the field of fODF peaks, and all subjects mentioned in the following are in this field.

A Registration-And Uncertainty-Based Framework For White Matter Tract Segmentation With Only One Annotated Subject

Abstract: White matter (WM) tract segmentation based on diffusion magnetic resonance imaging (dMRI) plays an important role in the analysis of human health and brain diseases. However, the annotation of WM tracts is time-consuming and needs experienced neuroanatomists. In this study, to explore tract segmentation in the challenging setting of minimal annotations, we propose a novel framework utilizing only one annotated subject (subject-level one-shot) for tract segmentation. Our method is constructed by proposed registration-based peak augmentation (RPA) and uncertaintybased refining (URe) modules. RPA module synthesizes pseudo subjects and their corresponding labels to improve the tract segmentation performance. The proposed URe module alleviates the negative influence of the low-confidence voxels on pseudo subjects. Experimental results show that our method outperforms other state-of-the-art methods by a large margin, and our proposed modules are effective. Overall, our method achieves accurate whole-brain tract segmentation with only one annotated subject. Our code is available at https://github.com/HaoXu0507/ ISBI2023-One-Shot-WM-Tract-Segmentation.

Tractseg - Fast And Accurate White Matter Tract Segmentation

Abstract: The individual course of white matter fiber tracts is an important factor for analysis of white matter characteristics in healthy and diseased brains. Diffusion-weighted MRI tractography in combination with region-based or clustering-based selection of streamlines is a unique combination of tools which enables the in-vivo delineation and analysis of anatomically well-known tracts. This, however, currently requires complex, computationally intensive processing pipelines which take a lot of time to set up. TractSeg is a novel convolutional neural network-based approach that directly segments tracts in the field of fiber orientation distribution function (fODF) peaks without using tractography, image registration or parcellation. We demonstrate that the proposed approach is much faster than existing methods while providing unprecedented accuracy, using a population of 105 subjects from the Human Connectome Project. We also show initial evidence that TractSeg is able to generalize to differently acquired data sets for most of the bundles. The code and data are openly available at https://github.com/MIC-DKFZ/TractSeg/ and https://doi.org/10.5281/zenodo.1088277 , respectively.


As per convention, each sh i point also contains a two-dimensional binary pen state #REFR -(1, 0): stroke is being drawn, (0, 1): the end of the stroke, (0, 0): padding points to account for the fact that all strokes have a different number of points.

In this section, we first introduce our classification model which is designed around strokes as sketch building blocks. We then introduce our method for model explainability.

As a pre-processing step, we simplify all sketches by the RDP algorithm #OTHEREFR .

For each stroke s i consisting of k points, {s i,1 , s i,2 , ..., s i,k−1 , s i,k }, we identify three inherent properties in s i and learn respective descriptor for each: location l i , shape sh i and stroke order o i .

We use the coordinate value (x i , y i ) of s i,1 to represent the location of each stroke l i .

In order to disentangle shape information sh i from its actual location, we use relative instead of absolute coordinates and move the starting point of all strokes to the canvas centre.

Our proposed sketch encoder f w , which we name SketchXAINet ("X" for EXplainability), first learns to encode l i , sh i and o i with different learnable components before fusing them together into a Transformer for final decision.

This tailored model design is then ready to undertake the novel explainability task defined later. A full high-level schematic is shown in Fig. 2 .

We use a bidirectional LSTM #OTHEREFR to extract shape information of each stroke sh i , and one linear layer for location l i embedding learning.

We pre-define the maximum number of strokes allowed and assign a learnable embedding for each order (time) embedding o i .

Sketchxai: A First Look At Explainability For Human Sketches

Abstract: Iterative Figure 1. Explainability, but for human sketches. We demonstrate a new methodology for explaining AI decisions on human sketch data. Instead of one static explanation per instance as in existing works, our proposed method supports generating infinitely many dynamic explanations with each showcasing how an AI classifier responds to a different view of the same sketch input. This is made possible by seeing stroke as the fundamental block to explain a sketch and randomising every stroke location to introduce stochasticity at every case. The infinite varieties of explanation paths allows humans to enjoy a wider coverage on how AI functions, and therefore to better scrutinise AI.

A Neural Representation Of Sketch Drawings

Abstract: We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format.


SMC-2 has better performance results on Cifar10/100 when compared to other competitors (LSR, DLB) #REFR .

From Table 1 , it can be seen that the improvement of SMC-2 on Cifar100 relative to vanilla its validation top-1 accuracy ranges from 0.74% to 1.87%, and on Cifar10 compared to vanilla its validation top-1 accuracy ranges from 0.26% to 0.65%, which shows that SMC-2 is effective.

Also compared to the advanced method SAM, our algorithm converges faster and better except for the WRN-28-10 model.

As can be seen in Figure 3 , SMC-2 is still effective on large data sets.

Compared with SGD, SMC-2 significantly improves the convergence speed and generalization ability.

However, due to the limitation of our computational resources, we cannot choose the hyperparameters of SMC-2 suitable for ImageNet, so we cannot compare it with other methods.

Figure 3 : Comparison of the validation Top-1 accuracy results of SGD and SMC-2 trained on ImageNet with AlexNet model using different seeds.

Self-Discipline On Multiple Channels

Abstract: Self-distillation relies on its own information to improve the generalization ability of the model and has a bright future. Existing self-distillation methods either require additional models, model modification, or batch size expansion for training, which increases the difficulty of use, memory consumption, and computational cost. This paper developed Self-discipline on multiple channels(SMC), which combines consistency regularization with self-distillation using the concept of multiple channels. Conceptually, SMC consists of two steps: 1) each channel data is simultaneously passed through the model to obtain its corresponding soft label, and 2) the soft label saved in the previous step is read together with the soft label obtained from the current channel data through the model to calculate the loss function. SMC uses consistent regularization and self-distillation to improve the generalization ability of the model and the robustness of the model to noisy labels. We named the SMC containing only two channels as SMC-2. Comparative experimental results on both datasets show that SMC-2 outperforms Label Smoothing Regularizaion and Self-distillation From The Last Mini-batch on all models, and outperforms the state-of-the-art Sharpness-Aware Minimization method on 83% of the models.Compatibility of SMC-2 and data augmentation experimental results show that using both SMC-2 and data augmentation improves the generalization ability of the model between 0.28% and 1.80% compared to using only data augmentation. Ultimately, the results of the label noise interference experiments show that SMC-2 curbs the tendency that the model's generalization ability decreases in the late training period due to the interference of label noise.

Variational Information Distillation For Knowledge Transfer

Abstract: Transferring knowledge from a teacher neural network pretrained on the same or a similar task to a student neural network can significantly improve the performance of the student neural network. Existing knowledge transfer approaches match the activations or the corresponding hand-crafted features of the teacher and the student networks. We propose an information-theoretic framework for knowledge transfer which formulates knowledge transfer as maximizing the mutual information between the teacher and the student networks. We compare our method with existing knowledge transfer methods on both knowledge distillation and transfer learning tasks and show that our method consistently outperforms existing methods. We further demonstrate the strength of our method on knowledge transfer across heterogeneous network architectures by transferring knowledge from a convolutional neural network (CNN) to a multi-layer perceptron (MLP) on CIFAR-10. The resulting MLP significantly outperforms the-state-of-the-art methods and it achieves similar performance to the CNN with a single convolutional layer.


Our work is perhaps most similar to #REFR , who propose a VAE on the latent space of another VAE learned directly on pixels.

(2022) also trains an encoder with the same loss objectives, but it outputs continuous values and trains a diffusion model over these latents.

Our work adapts the pretrained autoencoders from #OTHEREFR since our VAEs act in continuous space.

An alternative two-stage approach uses an upsampling stack which first creates low-resolution RGB images before upsampling them to a higher resolution #OTHEREFR .

This method works very well for diffusion models #OTHEREFR as diffusion models have good super-resolution capabilities #OTHEREFR .

In our hierarchical VAE experiments, we found that upsampling stack is outperformed by a latent-space approach (Section 4.3).

This work is similarly motivated to ours, accounting for the fact that the ambient dimension is much greater than the manifold dimension.

However, samples from their model are blurry since their first stage is a regular VAE trained with a pixel-space distortion term.

In contrast, we learn the underlying manifold using a DAE augmented with perceptual and aversarial losses, and achieve much higher compression rates with little to no blurriness.

Furthermore, our second stage is based off hierarchical VAEs instead of single-group ones to reap the benefits of high stochastic depth observed in #OTHEREFR .

Perceptual and Adversarial Losses in VAEs.

High Fidelity Image Synthesis With Deep Vaes In Latent Space

Abstract: We present fast, realistic image generation on high-resolution, multimodal datasets using hierarchical variational autoencoders (VAEs) trained on a deterministic autoencoder's latent space. In this two-stage setup, the autoencoder compresses the image into its semantic features, which are then modeled with a deep VAE. With this method, the VAE avoids modeling the fine-grained details that constitute the majority of the image's code length, allowing it to focus on learning its structural components. We demonstrate the effectiveness of our two-stage approach, achieving a FID of 9.34 on the ImageNet-256 dataset which is comparable to BigGAN. We make our implementation available online 1 .

Diagnosing And Enhancing Vae Models

Abstract: Although variational autoencoders (VAEs) represent a widely influential deep generative model, many aspects of the underlying energy function remain poorly understood. In particular, it is commonly believed that Gaussian encoder/decoder assumptions reduce the effectiveness of VAEs in generating realistic samples. In this regard, we rigorously analyze the VAE objective, differentiating situations where this belief is and is not actually true. We then leverage the corresponding insights to develop a simple VAE enhancement that requires no additional hyperparameters or sensitive tuning. Quantitatively, this proposal produces crisp samples and stable FID scores that are actually competitive with a variety of GAN models, all while retaining desirable attributes of the original VAE architecture. A shorter version of this work will appear in the ICLR 2019 conference proceedings (Dai and Wipf, 2019). The code for our model is available at https://github.com/daib13/ TwoStageVAE.


C. Dynamically Guided Learning Algorithm

L iou−v and L iou−t are the GIoU loss #REFR of the visible branch and thermal branch.

The α, β, and γ are set to 100, 1, and 100000 respectively.

Similarly, the total loss of the visible and thermal infrared branches can be described as follows:

where L dgl f v and L dgl f t are the dynamically guided learning loss of the visible branch and thermal branch guided by the fused branch, respectively.

Here, the settings of α, β and γ are the same as in Equation 9.

Note that, we use an iterative training strategy to avoid mutual interference among branches, i.e., the gradients of other branches will be truncated when optimizing a particular branch.

A similar training strategy is also used for the optimization of GAN network #OTHEREFR , #OTHEREFR .

Rgbt Tracking Via Progressive Fusion Transformer With Dynamically Guided Learning

Abstract: Existing Transformer-based RGBT tracking methods either use cross-attention to fuse the two modalities, or use self-attention and cross-attention to model both modality-specific and modality-sharing information. However, the significant appearance gap between modalities limits the feature representation ability of certain modalities during the fusion process. To address this problem, we propose a novel Progressive Fusion Transformer called ProFormer, which progressively integrates single-modality information into the multimodal representation for robust RGBT tracking. In particular, ProFormer first uses a self-attention module to collaboratively extract the multimodal representation, and then uses two cross-attention modules to interact it with the features of the dual modalities respectively. In this way, the modality-specific information can well be activated in the multimodal representation. Finally, a feed-forward network is used to fuse two interacted multimodal representations for the further enhancement of the final multimodal representation. In addition, existing learning methods of RGBT trackers either fuse multimodal features into one for final classification, or exploit the relationship between unimodal branches and fused branch through a competitive learning strategy. However, they either ignore the learning of single-modality branches or result in one branch failing to be well optimized. To solve these problems, we propose a dynamically guided learning algorithm that adaptively uses well-performing branches to guide the learning of other branches, for enhancing the representation ability of each branch. Extensive experiments demonstrate that our proposed ProFormer sets a new state-of-the-art performance on RGBT210, RGBT234, LasHeR, and VTUAV datasets. In particular, the improvements of ProFormer on the LasHeR dataset are up to +6.2%/ + 4.1% in PR/SR compared with existing best RGBT tracker.

Generalized Intersection Over Union: A Metric And A Loss For Bounding Box Regression

Abstract: Intersection over Union (IoU) is the most popular evaluation metric used in the object detection benchmarks. However, there is a gap between optimizing the commonly used distance losses for regressing the parameters of a bounding box and maximizing this metric value. The optimal objective for a metric is the metric itself. In the case of axis-aligned 2D bounding boxes, it can be shown that IoU can be directly used as a regression loss. However, IoU has a plateau making it infeasible to optimize in the case of non-overlapping bounding boxes. In this paper, we address the this weakness by introducing a generalized version of IoU as both a new loss and a new metric. By incorporating this generalized IoU ( GIoU) as a loss into the state-of-the art object detection frameworks, we show a consistent improvement on their performance using both the standard, IoU based, and new, GIoU based, performance measures on popular object detection benchmarks such as PASCAL VOC and MS COCO.


Meanwhile, we can see Ours-Ret achieves better results on TU-Berlin Ext and Sketchy Ext datasets, and works slightly worse than Ours-RN on Sketchy Ext #REFR Split and QuickDraw Ext.

ViT-Ret means replacing the class token in ViT with a retrieval token used for matching; while ViT-Vis uses the visual tokens for matching.

It should be noted that all the baselines, except CC-DG #OTHEREFR , StyleGuide #OTHEREFR and ViT variants #OTHEREFR , employ external semantic information, whereas our method only relies on the learned vi-sual correspondences between sketch-photo pairs.

We also compare two variants of our model, i.e., Ours-Ret and Ours-RN, which retrieve images based on the retrieval token and the relation network separately. Evaluation protocol.

Mean average precision (mAP), precision on top 100 (Prec@100) and top 200 (Prec@200) are reported following the standard evaluation protocol. Results.

From Table 1 , we can see that our proposed method achieves better or comparable results over other competitors, which is especially noteworthy since our method does not benefit from extra semantic information, i.e., text or class label.

Some qualitative results (including some failure cases) are illustrated in Figure 3 .

We can observe that, compared with SAKE and DSN, most of the top images retrieved using our approach more faithfully resemble to the query sketches in terms of the overall object pose and shape characteristics.

And the false positives given by our method are somehow reasonable, as they are superficially similar to the query. We additionally compare ours with baselines for generalized ZS-SBIR.

Results in Table 2 show that ours clearly outperforms others, suggesting a strong generalizability of our model. Ablation study.

An ablative study is conducted to examine the importance of each key component in our model.

Zero-Shot Everything Sketch-Based Image Retrieval, And In Explainable Style

Abstract: This paper studies the problem of zero-short sketchbased image retrieval (ZS-SBIR), however with two significant differentiators to prior art (i) we tackle all variants (inter-category, intra-category, and cross datasets) of ZS-SBIR with just one network ("everything"), and (ii) we would really like to understand how this sketch-photo matching operates ("explainable"). Our key innovation lies with the realization that such a cross-modal matching problem could be reduced to comparisons of groups of key local patches-akin to the seasoned "bag-of-words" paradigm. Just with this change, we are able to achieve both of the aforementioned goals, with the added benefit of no longer requiring external semantic knowledge. Technically, ours is a transformer-based cross-modal network, with three novel components (i) a self-attention module with a learnable tokenizer to produce visual tokens that correspond to the most informative local regions, (ii) a cross-attention module to compute local correspondences between the visual tokens across two modalities, and finally (iii) a kernel-based relation network to assemble local putative matches and produce an overall similarity metric for a sketch-photo pair. Experiments show ours indeed delivers superior performances across all ZS-SBIR settings. The all important explainable goal is elegantly achieved by visualizing crossmodal token correspondences, and for the first time, via sketch to photo synthesis by universal replacement of all matched photo patches. Code and model are available at https://github.com/buptLinfy/ZSE-SBIR.

A Zero-Shot Framework For Sketch-Based Image Retrieval

Abstract: Sketch-based image retrieval (SBIR) is the task of retrieving images from a natural image database that correspond to a given hand-drawn sketch. Ideally, an SBIR model should learn to associate components in the sketch (say, feet, tail, etc.) with the corresponding components in the image having similar shape characteristics. However, current evaluation methods simply focus only on coarse-grained evaluation where the focus is on retrieving images which belong to the same class as the sketch but not necessarily having the same shape characteristics as in the sketch. As a result, existing methods simply learn to associate sketches with classes seen during training and hence fail to generalize to unseen classes. In this paper, we propose a new benchmark for zero-shot SBIR where the model is evaluated in novel classes that are not seen during training. We show through extensive experiments that existing models for SBIR that are trained in a discriminative setting learn only class specific mappings and fail to generalize to the proposed zero-shot setting. To circumvent this, we propose a generative approach for the SBIR task by proposing deep conditional generative models that take the sketch as an input and fill the missing information stochastically. Experiments on this new benchmark created from the "Sketchy" dataset, which is a large-scale database of sketch-photo pairs demonstrate that the performance of these generative models is significantly better than several state-of-the-art approaches in the proposed zero-shot framework of the coarse-grained SBIR task.


Specifically, we first use the RoBERTa transformer #REFR to extract the vector embeddings of both claims and tweets, and then we compute the cosine similarity between each claim and tweet in our data, retaining the top-100 most similar tweets for each claim.

Given the collected fact-checked information, our goal is to find tweets reporting such verified claims.

However, as manually annotating the whole corpus of tweets in our dataset is not a suitable solution, we deploy a machine learning-based annotation strategy to maximize the likelihood of finding tweets related to one of the claims under analysis.

Consequently, we end up with 8,300 unique tweet-claim pairs that were inspected via a manual labeling process.

It is worth noting that the choice of the RoBERTa transformer depends on the higher similarity scores provided by this model with respect to other transformers (ms-marco-MiniLM-L-4-v2 and quora-roberta-base), which allows us to maximize the chance of finding a matching pair.

Next, we annotate the above-mentioned tweet-claim pairs considering a strict definition of relevance, i.e., the tweet discusses the claim if it explicitly mentions the same entities and events reported in the fact-checked information.

Table 1 shows some examples of false claims and matched tweets, which highlight how the matched tweet can discuss a claim without expressing any stance (examples no.

1 and 2) or can support or refute the claim (examples no. 3 and 4, respectively).

Retrieving False Claims On Twitter During The Russia-Ukraine Conflict

Abstract: Nowadays, false and unverified information on social media sway individuals' perceptions during major geo-political events and threaten the quality of the whole digital information ecosystem. Since the Russian invasion of Ukraine, several fact-checking organizations have been actively involved in verifying stories related to the conflict that circulated online. In this paper, we leverage a public repository of fact-checked claims to build a methodological framework for automatically identifying false and unsubstantiated claims spreading on Twitter in February 2022. Our framework consists of two sequential models: First, the claim detection model identifies whether tweets incorporate a (false) claim among those considered in our collection. Then, the claim retrieval model matches the tweets with fact-checked information by ranking verified claims according to their relevance with the input tweet. Both models are based on pre-trained language models and fine-tuned to perform a text classification task and an information retrieval task, respectively. In particular, to validate the effectiveness of our methodology, we consider 83 verified false claims that spread on Twitter during the first week of the invasion, and manually annotate 5,872 tweets according to the claim(s) they report. Our experiments show that our proposed methodology outperforms standard baselines for both claim detection and claim retrieval. Overall, our results highlight how social media providers could effectively leverage semi-automated * V. La Gatta and C. Wei contributed equally to this work.

Roberta: A Robustly Optimized Bert Pretraining Approach

Abstract: Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.