For example, we may use a generator architecture based on a styletransfer network which improves the generated sample quality #REFR .

In our whistle extraction experiments, we did not use this extended idea, because the appearance of our foreground object (the whistles) is relatively simple, i.e., the variance of appearance is mainly rooted in the whistle contour shape and whistle magnitude.

Therefore, we directly add whistle signals to the background using the third GAN in our framework.

Our framework can be readily extended to extract calls of other whale species (e.g, baleen whales) and to other similar tasks (e.g., semantic image segmentation).

Though it may not affect the main contributions of this work, our data generation method can be improved in three aspects in the future.

Firstly, we may use improved generative neural network architecture and training strategies.

The discriminator augmentation mechanism proposed in #OTHEREFR may help stabilize training in limited data regimes.

We may also explore generating larger patches of high quality with the method in #OTHEREFR .

Secondly, we may use real data in the data generation process to enrich the data variance.

The real background and annotated whistle contours can be used as the input data of our GAN in the third stage, and we can generate whistle signsals of novel shapes on real background or generate whistle signals of annotated contour shapes on GANgenerated background. Thirdly, we may improve the sample selection method.

In this paper, we use a simple yet effective pixel-wise entropy method to select whistle contour of good quality.

Learning Stage-Wise Gans For Whistle Extraction In Time-Frequency Spectrograms

Abstract: Whistle contour extraction aims to derive animal whistles from time-frequency spectrograms as polylines. For toothed whales, whistle extraction results can serve as the basis for analyzing animal abundance, species identity, and social activities. During the last few decades, as long-term recording systems have become affordable, automated whistle extraction algorithms were proposed to process large volumes of recording data. Recently, a deep learning-based method demonstrated superior performance in extracting whistles under varying noise conditions. However, training such networks requires a large amount of labor-intensive annotation, which is not available for many species. To overcome this limitation, we present a framework of stage-wise generative adversarial networks (GANs), which compile new whistle data suitable for deep model training via three stages: generation of background noise in the spectrogram, generation of whistle contours, and generation of whistle signals. By separating the generation of different components in the samples, our framework composes visually promising whistle data and labels even when few expert annotated data are available. Regardless of the amount of human-annotated data, the proposed data augmentation framework leads to a consistent improvement in performance of the whistle extraction model, with a maximum increase of 1.69 in the whistle extraction mean F1-score. Our stage-wise GAN also surpasses one single GAN in improving whistle extraction models with augmented data.

A Style-Based Generator Architecture For Generative Adversarial Networks

Abstract: We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.


We adopt soft-actor critic (SAC) #REFR as our policy optimization algorithm, which demonstrates impressive performance for continuous tasks.

π * = arg max π E ∞ t=0 r (s t , a t ) .

In maximum entropy reinforcement learning, the expected entropy term is augmented #OTHEREFR , which helps the agents explore more widely and give up on clearly unpromising avenues.

In this case, the optimal policy π * with parameter α that determines the relative importance of the expected entropy term can be written as:

π * = arg max π E ∞ t=0 [r (s t , a t ) + αH (π ( • |s t ))] . (2)

In the policy improvement step, the policy parameters can be directly updated by minimizing the following:

where φ represents parameters of policy network π φ (a t |s t ) and θ are parameters of soft Q-function Q θ (s t , a t ), D is the distribution of states and actions sampled from interaction experience, and Z θ (s t ) is the partition function that normalizes the distribution.

For our offline scenario, D is the distribution of the origin offline dataset combined with a replay buffer which contains simulated trajectories generated by the learned model.

Finally, actions a t are generated by re-parameterizing the policy using noise vector t :

Entropy: Environment Transformer And Offline Policy Optimization

Abstract: Model-based methods provide an effective approach to offline reinforcement learning (RL). They learn an environmental dynamics model from interaction experiences and then perform policy optimization based on the learned model. However, previous model-based offline RL methods lack long-term prediction capability, resulting in large errors when generating multi-step trajectories. We address this issue by developing a sequence modeling architecture, Environment Transformer, which can generate reliable long-horizon trajectories based on offline datasets. We then propose a novel model-based offline RL algorithm, ENTROPY, that learns the dynamics model and reward function by ENvironment TRansformer and performs Offline PolicY optimization. We evaluate the proposed method on MuJoCo continuous control RL environments. Results show that ENTROPY performs comparably or better than the state-of-the-art model-based and model-free offline RL methods and demonstrates more powerful long-term trajectory prediction capability compared to existing model-based offline methods.

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning With A Stochastic Actor

Abstract: Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy - that is, succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.

policy optimization algorithm
soft-actor critic


This is contrastive to the prior literature revealing that poor calibration of large language models often renders the models to rarely be uncertain of their outcomes #REFR .

The domain transfer from both tasks (or a combination of them) gives the active annotation a head-start for initial sample selection.

Table 3 shows the overlap of samples picked out in each iteration from the same larger data pool for the model at iteration 0 (base model).

RAN-DOM has the lowest overlaps with all the other strategies.

We also find that there is a significant overlap (> 80%) in the samples between ENTROPY or CAL, the state-of-the-art approach, and PRC.

CAL has a higher overlap with ENTROPY rather than CORESET, showing that samples deemed to be both highly informative and contrastive by the model are also usually likely to be dissonant.

All strategies except RANDOM have > 55% overlap with each other.

This implies that diversity-and uncertainty-based methods are not as different from each other as they theoretically are and inclined to pick similar samples -hinting that a lot of diversity-based sampling measures mostly pick highly informative samples as well.

Furthermore, PRC tends to choose samples that the "state-of-the-art" model also picks in rare-case scenarios, indicating that it could be a computationally inexpensive alternative.

Table 4 shows the results averaged over two rounds of active annotation and learning for five strategies with two types of model updates.

While the performance for dissonance class across all strategies do not seem to boost much in a single round of active learning (since adding 300 new annotations adds only between 10-30 dissonance examples in each round), Figure 3 shows that the CM approach always performs better than IT.

Transfer And Active Learning For Dissonance Detection: Addressing The Rare-Class Challenge

Abstract: While transformer-based systems have enabled greater accuracies with fewer training examples, data acquisition obstacles still persist for rare-class tasks-when the class label is very infrequent (e.g. < 5% of samples). Active learning has in general been proposed to alleviate such challenges, but choice of selection strategy, the criteria by which rareclass examples are chosen, has not been systematically evaluated. Further, transformers enable iterative transfer-learning approaches. We propose and investigate transfer-and active learning solutions to the rare class problem of dissonance detection through utilizing models trained on closely related tasks and the evaluation of acquisition strategies, including a proposed probability-of-rare-class (PRC) approach. We perform these experiments for a specific rare class problem: collecting language samples of cognitive dissonance from social media. We find that PRC is a simple and effective strategy to guide annotations and ultimately improve model accuracy while transferlearning in a specific order can improve the cold-start performance of the learner but does not benefit iterations of active learning.

On Calibration Of Modern Neural Networks

Abstract: Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.


The Multimodal EmotionLines Dataset (MELD) #REFR is an example of Emotion Recognition in Conversations (ERC) and attempts to estimate the emotion for every dialog utterance in TV episodes from Friends.

Different from the above, our work focuses on multi-label emotions and mental states recognition in movies exploiting multimodal context both at the scene-and character-level.

Multimodal datasets for emotion recognition have seen recent adoption.

Acted Facial Expressions in the Wild #OTHEREFR aims to predict emotions from faces, but does not provide any context.

The Stanford Emotional Narratives Dataset #OTHEREFR contains participant shared narratives of positive/negative events in their lives.

While multimodal, these are quite different from edited movies and stories that are our focus.

Different from MELD, we operate at the time-scale of a cohesive story unit, a movie scene.

Finally, closest to our work, Annotated Creative Commons Emotional DatabasE (LIRIS-ACCEDE) #OTHEREFR obtains emotion annotations for short movie clips.

However, the clips are quite small (8-12s) and annotations are obtained in the continuous valence-arousal space.

Different from the above works, we also aim to predict character-level mental states and demonstrate that video and dialog context helps for such labels.

How You Feelin'? Learning Emotions And Mental States In Movie Scenes

Abstract: Movie story analysis requires understanding characters' emotions and mental states. Towards this goal, we formulate emotion understanding as predicting a diverse and multi-label set of emotions at the level of a movie scene and for each character. We propose EmoTx, a multimodal Transformer-based architecture that ingests videos, multiple characters, and dialog utterances to make joint predictions. By leveraging annotations from the MovieGraphs dataset [74], we aim to predict classic emotions (e.g. happy, angry) and other mental states (e.g. honest, helpful). We conduct experiments on the most frequently occurring 10 and 25 labels, and a mapping that clusters 181 labels to 26. Ablation studies and comparison against adapted stateof-the-art emotion recognition approaches shows the effectiveness of EmoTx. Analyzing EmoTx's self-attention scores reveals that expressive emotions often look at character tokens while other mental states rely on video and dialog cues.

Meld: A Multimodal Multi-Party Dataset For Emotion Recognition In Conversations

Abstract: Emotion recognition in conversations is a challenging Artificial Intelligence (AI) task. Recently, it has gained popularity due to its potential applications in many interesting AI tasks such as empathetic dialogue generation, user behavior understanding, and so on. To the best of our knowledge, there is no multimodal multi-party conversational dataset available, which contains more than two speakers in a dialogue. In this work, we propose the Multimodal EmotionLines Dataset (MELD), which we created by enhancing and extending the previously introduced EmotionLines dataset. MELD contains 13,708 utterances from 1433 dialogues of Friends TV series. MELD is superior to other conversational emotion recognition datasets SEMAINE and IEMOCAP as it consists of multiparty conversations and number of utterances in MELD is almost twice as these two datasets. Every utterance in MELD is associated with an emotion and a sentiment label. Utterances in MELD are multimodal encompassing audio and visual modalities along with the text. We have also addressed several shortcomings in EmotionLines and proposed a strong multimodal baseline. The baseline results show that both contextual and multimodal information play important role in emotion recognition in conversations.


For example, we refer to the study by Esfahani and Kuhn #REFR , where the ambiguity set is formed by a ball in the space of (multivariate and non-discrete) probability distributions with respect to the Wasserstein metric.

As the first example, we refer to the study by Delage and Ye #OTHEREFR , who consider convex optimization problems with random parameters in the objective function.

The ambiguity set in #OTHEREFR is constructed by leveraging a given convex support and confidence sets for the mean and the covariance matrix of the uncertain problem parameters. Then it is

shown that for several particular forms of the loss function (x, c), the proposed distributionally robust formulation can be solved efficiently by leveraging strong duality results for moment problems #OTHEREFR .

Despite the fact that moment-based ambiguity sets are rather standard in the DRO literature; see, e.g., #OTHEREFR , it is demonstrated in #OTHEREFR that using the second-order moment constraints with a linear mixed-integer set of feasible decisions X results in a non-linear single-level MIP reformulation, which can only be approximated by a sequence of semi-definite programming relaxations.

In this regard, several related studies focus on data-driven DRO problems, which admit linear MIP reformulations.

The center of the ball is at the uniform distribution on the training samples and the radius can be viewed as a decreasing function in the sample size.

The authors demonstrate that, if the Wasserstein metric is defined in terms of l 1 -norm or l ∞ -norm, then under some mild assumptions the associated worst-case expectation problem admits an equivalent dual linear programming reformulation.

Furthermore, whenever the set X is linear mixed-integer, the associated DRO problem can be recast as a linear MIP problem; see, e.g., #OTHEREFR .

On the other hand, Bertsimas et al. [6] analyze a data-driven robust optimization approach.

Instead of the stochastic programming formulation (1), the authors in #OTHEREFR consider an associated robust formulation of the form:

Distributionally Robust Mixed-Integer Programming With Wasserstein Metric: On The Value Of Uncertain Data

Abstract: This study addresses a class of linear mixed-integer programming (MIP) problems that involve uncertainty in the objective function coefficients. The coefficients are assumed to form a random vector, which probability distribution can only be observed through a finite training data set. Unlike most of the related studies in the literature, we also consider uncertainty in the underlying data set. The data uncertainty is described by a set of linear constraints for each random sample, and the uncertainty in the distribution (for a fixed realization of data) is defined using a type-1 Wasserstein ball centered at the empirical distribution of the data. The overall problem is formulated as a three-level distributionally robust optimization (DRO) problem. We prove that for a class of bi-affine loss functions the three-level problem admits a linear MIP reformulation. Furthermore, it turns out that in several important particular cases the three-level problem can be solved reasonably fast by leveraging the nominal MIP problem. Finally, we conduct a computational study, where the out-of-sample performance of our model and computational complexity of the proposed MIP reformulation are explored numerically for several application domains.

Data-Driven Distributionally Robust Optimization Using The Wasserstein Metric: Performance Guarantees And Tractable Reformulations

Abstract: We consider stochastic programs where the distribution of the uncertain parameters is only observable through a finite training dataset. Using the Wasserstein metric, we construct a ball in the space of (multivariate and non-discrete) probability distributions centered at the uniform distribution on the training samples, and we seek decisions that perform best in view of the worst-case distribution within this Wasserstein ball. The state-of-the-art methods for solving the resulting distributionally robust optimization problems rely on global optimization techniques, which quickly become computationally excruciating. In this paper we demonstrate that, under mild assumptions, the distributionally robust optimization problems over Wasserstein balls can in fact be reformulated as finite convex programs—in many interesting cases even as tractable linear programs. Leveraging recent measure concentration results, we also show that their solutions enjoy powerful finite-sample performance guarantees. Our theoretical results are exemplified in mean-risk portfolio optimization as well as uncertainty quantification.


Faster-ILOD #REFR adopts a similar approach to continual object detection than ILOD, using KD as the approach to limit forgetting.

However, rather than only distilling knowledge from the last layer, namely the class logits and box coordinates, the authors add a KD loss between the feature maps of the teacher and student networks, and another for the RPN.

This time, a L1 loss is used for distillation of feature maps activations as follows:

where H, W and D are the height, width and depth of the feature maps, respectively.

Notice that KD is only applied if the activation of the previous model is larger than the one of the current model.

Class-Incremental Learning Of Plant And Disease Detection: Growing Branches With Knowledge Distillation

Abstract: This paper investigates the problem of class-incremental object detection for agricultural applications where a model needs to learn new plant species and diseases incrementally without forgetting the previously learned ones. We adapt two public datasets to include new categories over time, simulating a more realistic and dynamic scenario. We then compare three class-incremental learning methods that leverage different forms of knowledge distillation to mitigate catastrophic forgetting. Our experiments show that all three methods suffer from catastrophic forgetting, but the recent Dynamic Y-KD approach, which additionally uses a dynamic architecture that grows new branches to learn new tasks, outperforms ILOD and Faster-ILOD in most scenarios both on new and old classes. These results highlight the challenges and opportunities of continual object detection for agricultural applications. In particular, the large intra-class and small interclass variability that is typical of plant images exacerbate the difficulty of learning new categories without interfering with previous knowledge. We publicly release our code to encourage future work. 1

Faster Ilod: Incremental Learning For Object Detectors Based On Faster Rcnn

Abstract: The human vision and perception system is inherently incremental where new knowledge is continually learned over time whilst existing knowledge is retained. On the other hand, deep learning networks are ill-equipped for incremental learning. When a well-trained network is adapted to new categories, its performance on the old categories will dramatically degrade. To address this problem, incremental learning methods have been explored to preserve the old knowledge of deep learning models. However, the state-of-the-art incremental object detector employs an external fixed region proposal method that increases overall computation time and reduces accuracy compared to object detectors such as Faster RCNN that use trainable Region Proposal Networks (RPNs). The purpose of this paper is to design an efficient end-to-end incremental object detector using knowledge distillation for object detectors based on RPNs. We first evaluate and analyze the performance of RPN-based detector with classic distillation towards incremental detection tasks. Then, we introduce multi-network adaptive distillation that properly retains knowledge from the old categories when fine-turning the model for new task. Experiments on the benchmark datasets, PASCAL VOC and COCO, demonstrate that the proposed incremental detector is more accurate as well as being 13 times faster than the baseline detector.


Prior work #REFR has demonstrated the advantages of performing audio-visual separation at the object level for videos with multiple objects.

To predict the separated audio source, each element of the mask is multiplied with the corresponding location in the input spectrogram: A S =M A S , where denotes the Hadamard product. Video and Multiple Instance Learning for mask prediction.

Since language annotations are not used during training, we train only with unlabeled videos.

During training, as well as inference on audio-visual separation, we use the CLIP visual encoder to extract a set of language-grounded spatiotemporal region representations f V ∈ R T ×H×W ×D .

Similarly, we can perform the same mask inference operation using videos by using the representation of each spatiotemporal region representation interchangeably in place of the language representation.

This interchangeability is due to the fact that the CLIP representations project the video to the joint vision-language embedding space.

In the absence of object detectors and labels, we propose an MIL formulation, where all spatiotemporal regions in the input video are treated as positive candidates.

Specifically, given a spatiotemporal grid of region representations extracted from the visual encoder as defined in this section, we predict a spectrogram mask M j for the j-th region.

Finally, we aggregate them over all regions to obtain a spectrogram mask for the entire video:

Intuitively, this formulation encourages the model to learn to identify the salient regions with sounding objects for audio separation.

Language-Guided Audio-Visual Source Separation Via Trimodal Consistency

Abstract: We propose a self-supervised approach for learning to perform audio source separation in videos based on natural language queries, using only unlabeled video and audio pairs as training data. A key challenge in this task is learning to associate the linguistic description of a soundemitting object to its visual features and the corresponding components of the audio waveform, all without access to annotations during training. To overcome this challenge, we adapt off-the-shelf vision-language foundation models to provide pseudo-target supervision via two novel loss functions and encourage a stronger alignment between the audio, visual and natural language modalities. During inference, our approach can separate sounds given text, video and audio input, or given text and audio input alone. We demonstrate the effectiveness of our self-supervised approach on three audiovisual separation datasets, including MUSIC, SOLOS and AudioSet, where we outperform state-of-the-art strongly supervised approaches despite not using object detectors or text labels during training. Our project page including publicly available code can be found at https://cs-people.bu.edu/rxtan/projects/VAST.

Co-Separating Sounds Of Visual Objects

Abstract: Learning how objects sound from video is challenging, since they often heavily overlap in a single audio channel. Current methods for visually-guided audio source separation sidestep the issue by training with artificially mixed video clips, but this puts unwieldy restrictions on training data collection and may even prevent learning the properties of "true" mixed sounds. We introduce a co-separation training paradigm that permits learning object-level sounds from unlabeled multi-source videos. Our novel training objective requires that the deep neural network's separated audio for similar-looking objects be consistently identifiable, while simultaneously reproducing accurate video-level audio tracks for each source training pair. Our approach disentangles sounds in realistic test videos, even in cases where an object was not observed individually during training. We obtain state-of-the-art results on visually-guided audio source separation and audio denoising for the MUSIC, AudioSet, and AV-Bench datasets.


While in this paper our focus is on the contractivity and dissipativity properties of the continuous-time model (1), we remark that integration methods that preserve contractivity have been proposed in #REFR .

To summarize, our goal is to learn the parameters θ of a dynamical system (1) -i.e., a Neural ODE -that optimize a given cost as per (2), with the hard constraint that either (3) or (6) (or both) hold.

This new requirement (i.e., the hard constraint) must be satisfied for all the parameters θ we optimize over.

In other words, we consider a form of failsafe learning, in the sense that the property of the model to be contracting or dissipative must be guaranteed during and after parameter optimization. Remark 1.

For solving problem (2), one has to select an integration scheme for (1), which allows performing forward propagation and gradient computations.

The properties of the continuous-time model (1) may not coincide with those of the discretized one, in general.

Tailoring these integration methods to IQC properties, or developing new ones, is an interesting direction for future work.

Incremental Property Q R S s ∆ (∆u, ∆y) L 2 -gain bound (γ ≥ 0) − 1 γ I γI 0 γ|∆u| 2 − 1 γ |∆y| 2 Passivity 0 0 1 2 I ∆u ∆y Input Passivity (ν ≥ 0) 0 −2νI I ∆u ∆y − ν|∆u| 2 Output Passivity (ε ≥ 0) −2εI 0 I ∆u ∆y − ε|∆y| 2

CONTRACTIVITY AND DISSIPATIVITY OF NODERENS In this section, we present and analyze novel DNN structures that arise from appropriately combining Neural ODEs #OTHEREFR with RENs #OTHEREFR .

The starting idea is to define the functions f θ and g θ in (1) through the model utilized for the layer equation in #OTHEREFR .

Unconstrained Parametrization Of Dissipative And Contracting Neural Ordinary Differential Equations

Abstract: In this work, we introduce and study a class of Deep Neural Networks (DNNs) in continuous-time. The proposed architecture stems from the combination of Neural Ordinary Differential Equations (Neural ODEs) with the model structure of recently introduced Recurrent Equilibrium Networks (RENs). We show how to endow our proposed NodeRENs with contractivity and dissipativity-crucial properties for robust learning and control. Most importantly, as for RENs, we derive parametrizations of contractive and dissipative NodeRENs which are unconstrained, hence enabling their learning for a large number of parameters. We validate the properties of NodeRENs, including the possibility of handling irregularly sampled data, in a case study in nonlinear system identification.

Contracting Nonlinear Observers: Convex Optimization And Learning From Data

Abstract: A new approach to design of nonlinear observers (state estimators) is proposed. The main idea is to (i) construct a convex set of dynamical systems which are contracting observers for a particular system, and (ii) optimize over this set for one which minimizes a bound on state-estimation error on a simulated noisy data set. We construct convex sets of continuous-time and discrete-time observers, as well as contracting sampled-data observers for continuous-time systems. Convex bounds for learning are constructed using Lagrangian relaxation. The utility of the proposed methods are verified using numerical simulation.


It was shown in #REFR that every cograph can be odd coloured using at most three colours, and that graphs of treewidth at most k can be odd coloured using at most k + 1 colours.

One way of Conjecture 3 is to consider that subdivided cliques appear to be essentially the graphs that require most colours to be odd coloured.

More specifically, consider the following family of graphs B = {G obtained from a graph G by adding, for every pair of vertices u, v ∈ V (G), a vertex w uv and edges uw uv and vw uv , and G has even order}.

Note that subdivided cliques of even order are exactly those graphs in B where graph G is edgeless, and that the graphs in B have χ odd (G ) = |V (G)| ∈ Θ( |V (G )|).

A question closely related to Conjecture 3 is whether if a class of graphs G does not contain arbitrarily large graphs of B as induced subgraphs, then G has odd chromatic number O( √ n), i.e., they satisfy Conjecture 3.

This question was already answered positively for some graph classes. In fact, the bounds provided were constant.

In fact, those results can easily be extended to all graphs admitting a join, and H-minor free graphs, respectively. Using a similar argument, Aashtab et al.

#OTHEREFR showed that planar graphs are 4-odd colourable, and this is tight due to subdivided K 4 being planar and 4-odd colourable, as explained above.

They also proved that subcubic graphs are 4-odd colourable, which is again tight due to subdivided K 4 , and conjecture that this result can be generalized to all graphs, i.e., χ odd (G) ≤ ∆ + 1, where ∆ denotes the maximum degree of G.

Observe that none of those graph classes contain arbitrarily large graphs from B as induced subgraphs.

On the negative side, bipartite graphs and split graphs contain arbitrarily large graphs from B, and therefore the bound of Conjecture 3 is best possible.

Odd Chromatic Number Of Graph Classes

Abstract: A graph is called odd (respectively, even) if every vertex has odd (respectively, even) degree. Gallai proved that every graph can be partitioned into two even induced subgraphs, or into an odd and an even induced subgraph. We refer to a partition into odd subgraphs as an odd colouring of G. Scott [Graphs and Combinatorics, 2001] proved that a graph admits an odd colouring if and only if it has an even number of vertices. We say that a graph G is k-odd colourable if it can be partitioned into at most k odd induced subgraphs. We initiate the systematic study of odd colouring and odd chromatic number of graph classes. In particular, we consider for a number of classes whether they have bounded odd chromatic number. Our main results are that interval graphs, graphs of bounded modular-width and graphs of bounded maximum degree all have bounded odd chromatic number. 2012 ACM Subject Classification Mathematics of computing → Graph coloring Keywords and phrases graph classes, vertex partition problem, odd colouring, colouring variant, upper bounds

On The Complexity Of Finding Large Odd Induced Subgraphs And Odd Colorings

Abstract: We study the complexity of the problems of finding, given a graph $G$, a largest induced subgraph of $G$ with all degrees odd (called an odd subgraph), and the smallest number of odd subgraphs that partition $V(G)$. We call these parameters ${\sf mos}(G)$ and $\chi_{\sf odd}(G)$, respectively. We prove that deciding whether $\chi_{\sf odd}(G) \leq q$ is polynomial-time solvable if $q \leq 2$, and NP-complete otherwise. We provide algorithms in time $2^{O({\sf rw})} \cdot n^{O(1)}$ and $2^{O(q \cdot {\sf rw})} \cdot n^{O(1)}$ to compute ${\sf mos}(G)$ and to decide whether $\chi_{\sf odd}(G) \leq q$ on $n$-vertex graphs of rank-width at most ${\sf rw}$, respectively, and we prove that the dependency on rank-width is asymptotically optimal under the ETH. Finally, we give some tight bounds for these parameters on restricted graph classes or in relation to other parameters.


We train NPP with ADAM #REFR and a batch size of 2, for a maximum of 60 epochs.

We created a large-scale dataset of 3D T1-weighted (T1w) brain MRI volumes by aggregating 7 datasets: GSP #OTHEREFR , ADNI #OTHEREFR , OASIS #OTHEREFR , ADHD #OTHEREFR , ABIDE #OTHEREFR , MCIC #OTHEREFR , and COBRE #OTHEREFR . The whole training set contains 10,083 scans.

As ground-truth target images, we used FreeSurfer generated skull-stripped, intensity-normalized and affine-registered (to so-called MNI atlas coordinates) images.

The initial learning rate is 1e-4 and then decreases by half after 30 epochs.

We use random gamma transformation as a data augmentation technique, with parameter log gamma (-0.3,0.3).

We randomly sampled λ from a log-uniform distribution on (−3, 1) for each mini-batch.

f θ is a U-Net-style architecture containing an encoder and decoder with skip connections in between.

Neural Pre-Processing: A Learning Framework For End-To-End Brain Mri Pre-Processing

Abstract: Head MRI pre-processing involves converting raw images to an intensity-normalized, skull-stripped brain in a standard coordinate space. In this paper, we propose an end-to-end weakly supervised learning approach, called Neural Pre-processing (NPP), for solving all three sub-tasks simultaneously via a neural network, trained on a large dataset without individual sub-task supervision. Because the overall objective is highly under-constrained, we explicitly disentangle geometricpreserving intensity mapping (skull-stripping and intensity normalization) and spatial transformation (spatial normalization). Quantitative results show that our model outperforms state-of-the-art methods which tackle only a single sub-task. Our ablation experiments demonstrate the importance of the architecture design we chose for NPP. Furthermore, NPP affords the user the flexibility to control each of these tasks at inference time. The code and model are freely-available at https: //github.com/Novestars/Neural-Pre-processing.

Adam: A Method For Stochastic Optimization

Abstract: We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.


The error achieves 16cm for incidence angles around 80°which confirms the results described in #REFR for another lidar sensor.

The goal is estimation of the difference between the measured board pose with respect to its ground truth for different orientations.

The truth board pose is provided by the theodolite sensor that tracks board's top-left corner.

Knowing the board's dimensions and the fact that its base is fixed, the full pose of the board is given.

As the metric, we calculate the point-to-point distance between the board's measured point cloud and the point cloud sampled from its mesh (1000 points are sampled uniformly from the board's cuboid).

The metric's values for different board angles are given in the Figure 5 .

As dark surfaces have weaker reflective qualities, it was observed that the error is larger when the black side of the board was measured.

The metric values for higher angles are not provided as there are almost no points measured by the lidar of the board oriented almost perpendicular.

Note, that we did not observe dependence of the ICP point-to-point distance on incidence angles when the white-side target was placed reasonably close to the lidar (for 5.3m distance).

Self-Supervised Depth Correction Of Lidar Measurements From Map Consistency Loss

Abstract: Depth perception is considered an invaluable source of information in the context of 3D mapping and various robotics applications. However, point cloud maps acquired using consumer-level light detection and ranging sensors (lidars) still suffer from bias related to local surface properties such as measuring beam-to-surface incidence angle, distance, texture, reflectance, or illumination conditions. This fact has recently motivated researchers to exploit traditional filters, as well as the deep learning paradigm, in order to suppress the aforementioned depth sensors error while preserving geometric and map consistency details. Despite the effort, depth correction of lidar measurements is still an open challenge mainly due to the lack of clean 3D data that could be used as ground truth. In this paper, we introduce two novel point cloud map consistency losses, which facilitate self-supervised learning on real data of lidar depth correction models. Specifically, the models exploit multiple point cloud measurements of the same scene from different viewpoints in order to learn to reduce the bias based on the constructed map consistency signal. Complementary to the removal of the bias from the measurements, we demonstrate that the depth correction models help to reduce localization drift. Additionally, we release a data set that contains point cloud data captured in an indoor corridor environment with precise localization and ground truth mapping information 1 .

Lidar Measurement Bias Estimation Via Return Waveform Modelling In A Context Of 3D Mapping

Abstract: In a context of 3D mapping, it is very important to obtain accurate measurements from sensors. In particular, Light Detection And Ranging (LIDAR) measurements are typically treated as a zero-mean Gaussian distribution. We show that this assumption leads to predictable localisation drifts, especially when a bias related to measuring obstacles with high incidence angles is not taken into consideration. Moreover, we present a way to physically understand and model this bias, which generalizes to multiple sensors. Using an experimental setup, we measured the bias of the Sick LMS151, Velodyne HDL-32E, and Robosense RS-LiDAR-16 as a function of depth and incidence angle, and showed that the bias can reach 20 cm for high incidence angles. We then used our model to remove the bias from the measurements, leading to more accurate maps and a reduced localisation drift.


Is IBMIL just post-processing? Since our proposed IB-MIL shares some commonalities with the embedding-based MIL #REFR , one may ask: Do the improvements only come from exploring the bag relations? To answer this question, we make minor modifications on ABMIL.

We will explore the way of incorporating bag labels in future work.

On the other hand, the unsupervised fashion makes our scheme compatible with large-scale unlabelled datasets.

We explore more unlabelled bags via combining the bags of TCGA and Camelyon16, and constituting the confounder set via the non-parametric aggregators. From Fig.

5b , we observe a clear improvement on AUC under both max-and mean-poolings.

That indicates, with more bags, our implementation can achieve better approximation of confounders.

Instead of interventional training with confounders, we obtain the confounder dictionary via the class-specific K-means and treat it as a KNN classifier for evaluation.

As can be seen, it brings limited improvements or even degrades the performance, verifying the improvements comes from interventional training, which is not just post-processing.

Interventional Bag Multi-Instance Learning On Whole-Slide Pathological Images

Abstract: Multi-instance learning (MIL) is an effective paradigm for whole-slide pathological images (WSIs) classification to handle the gigapixel resolution and slide-level label. Prevailing MIL methods primarily focus on improving the feature extractor and aggregator. However, one deficiency of these methods is that the bag contextual prior may trick the model into capturing spurious correlations between bags and labels. This deficiency is a confounder that limits the performance of existing MIL methods. In this paper, we propose a novel scheme, Interventional Bag Multi-Instance Learning (IBMIL), to achieve deconfounded bag-level prediction. Unlike traditional likelihood-based strategies, the proposed scheme is based on the backdoor adjustment to achieve the interventional training, thus is capable of suppressing the bias caused by the bag contextual prior. Note that the principle of IBMIL is orthogonal to existing bag MIL methods. Therefore, IBMIL is able to bring consistent performance boosting to existing schemes, achieving new state-of-the-art performance. Code is available at https://github.com/HHHedo/IBMIL.

Revisiting Multiple Instance Neural Networks

Abstract: Recently neural networks and multiple instance learning are both attractive topics in Artificial Intelligence related research fields. Deep neural networks have achieved great success in supervised learning problems, and multiple instance learning as a typical weakly-supervised learning method is effective for many applications in computer vision, biometrics, nature language processing, etc. In this paper, we revisit the problem of solving multiple instance learning problems using neural networks. Neural networks are appealing for solving multiple instance learning problem. The multiple instance neural networks perform multiple instance learning in an end-to-end way, which take a bag with various number of instances as input and directly output bag label. All of the parameters in a multiple instance network are able to be optimized via back-propagation. We propose a new multiple instance neural network to learn bag representations, which is different from the existing multiple instance neural networks that focus on estimating instance label. In addition, recent tricks developed in deep learning have been studied in multiple instance networks, we find deep supervision is effective for boosting bag classification accuracy. In the experiments, the proposed multiple instance networks achieve state-of-the-art or competitive performance on several MIL benchmarks. Moreover, it is extremely fast for both testing and training, e.g., it takes only 0.0003 second to predict a bag and a few seconds to train on a MIL datasets on a moderate CPU.


We also experimented with different types of feature extractors for SFG module, and observed that a Resnet18 or Resnet50 #REFR is more effective than a 2 linear layer feature extractor as shown in Table 2 .

Using CH 4 candidates feature from SFG improves the bounding box detection performance by 0.14 mAP and mask prediction by 0.09 mIOU.

This demonstrate that guiding queries with CH 4 candidates feature generated by SFG produces better embeddings as compared to random queries.

Along with this, we explored the provision of CH 4 candidates feature at 2 places, (i) at input level concatenating it with f comb ; and (ii) as input to query refiner.

We see an improvement of 0.09 mAP and 0.08 mIOU when SFG module output is passed to query refiner.

We hypothesize that this is because on concatenating with input features, the CH 4 candidates feature information gets lost, while as cross-attention with queries reduces the search space for decoder and generate better embeddings.

Spectral Linear Filter: We experimented with SLF for computing covariance (Cov) using different subset of columns in the input hyperspectral image.

We observed that the SLF is most effective when covariance is computed class-wise based on land cover.

Class-wise Cov ensures that the radiance absorption by ground terrain is same for all the pixels while computing CH 4 enhancement. As can be seen in Fig.

4 , SLF amplifies CH 4 candidate detection and reduces false positives.

SLF leads to a 0.03 mAP improved in detection compared to traditional filters. The prediction from MM is shown row-1 of Fig. 5 .

Methanemapper: Spectral Absorption Aware Hyperspectral Transformer For Methane Detection

Abstract: Methane (CH 4) is the chief contributor to global climate change. Recent Airborne Visible-Infrared Imaging Spectrometer-Next Generation (AVIRIS-NG) has been very useful in quantitative mapping of methane emissions. Existing methods for analyzing this data are sensitive to local terrain conditions, often require manual inspection from domain experts, prone to significant error and hence are not scalable. To address these challenges, we propose a novel end-to-end spectral absorption wavelength aware transformer network, MethaneMapper, to detect and quantify the emissions. MethaneMapper introduces two novel modules that help to locate the most relevant methane plume regions in the spectral domain and uses them to localize these accurately. Thorough evaluation shows that MethaneMapper achieves 0.63 mAP in detection and reduces the model size (by 5×) compared to the current state of the art. In addition, we also introduce a large-scale dataset of methane plume segmentation mask for over 1200 AVIRIS-NG flight lines from 2015-2022. It contains over 4000 methane plume sites. Our dataset will provide researchers the opportunity to develop and advance new methods for tackling this challenging greenhouse gas detection problem with significant broader social impact. Dataset and source code link 1 .

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


Thus, active block pairs of different rounds are stored interleaved in and the end of a round does not induce a synchronization point as in the original algorithm #REFR .

Threads that are not involved in scheduling can join parallel flow computations. In a parameter study [62, p.

108] , we found that = 1 offers a good trade-off between conflicting searches and scalability.

Initially, we push all pairs of adjacent blocks into a concurrent FIFO queue .

The threads then poll from and if a search finds an improvement on a block pair ( , ), we mark both as active using a seperate bitset for each round.

If either or becomes active, we push all adjacent blocks into if they are not contained yet.

A round ends when all of its block pairs have been processed and all prior rounds have ended.

If the relative improvement at the end of a round is less than 0.1%, we immediately terminate the algorithm.

t s t s ∞ ∞ ∞ ∞ ∞ ∞ ω(e) ein eout V 1 V 2 B 1 B 2

Since concurrently scheduled flow computations can operate on overlapping regions, there are three conflict types that can occur when applying a sequence of node moves to the global partition Π: balance constraint violations, Δ exp ≠ Δ (i.e., the expected does not match the actual connectivity reduction), and nodes in may already be moved by other searches.

Scalable High-Quality Hypergraph Partitioning

Abstract: Balanced hypergraph partitioning is an NP-hard problem with many applications, e.g., optimizing communication in distributed data placement problems. The goal is to place all nodes across k different blocks of bounded size, such that hyperedges span as few parts as possible. This problem is well-studied in sequential and distributed settings, but not in shared-memory. We close this gap by devising efficient and scalable shared-memory algorithms for all components employed in the best sequential solvers without compromises with regards to solution quality. This work presents the scalable and high-quality hypergraph partitioning framework Mt-KaHyPar. Its most important components are parallel improvement algorithms based on the FM algorithm and maximum flows, as well as a parallel clustering algorithm for coarsening-which are used in a multilevel scheme with log() levels. As additional components, we parallelize the-level partitioning scheme, devise a deterministic version of our algorithm, and present optimizations for plain graphs. We evaluate our solver on more than 800 graphs and hypergraphs, and compare it with 25 different algorithms from the literature. Our fastest configuration outperforms almost all existing hypergraph partitioners with regards to both solution quality and running time. Our highest-quality configuration achieves the same solution quality as the best sequential partitioner KaHyPar, while being an order of magnitude faster with ten threads. Thus, two of our configurations occupy all fronts of the Pareto curve for hypergraph partitioning. Furthermore, our solvers exhibit good speedups, e.g., 29.6x in the geometric mean on 64 cores (deterministic), 22.3x (log()-level), and 25.9x (-level).

Engineering Multilevel Graph Partitioning Algorithms

Abstract: We present a multi-level graph partitioning algorithm using novel local improvement algorithms and global search strategies transferred from multigrid linear solvers. Local improvement algorithms are based on max-flow min-cut computations and more localized FM searches. By combining these techniques, we obtain an algorithm that is fast on the one hand and on the other hand is able to improve the best known partitioning results for many inputs. For example, in Walshaw's well known benchmark tables we achieve 317 improvements for the tables at 1%, 3% and 5% imbalance. Moreover, in 118 out of the 295 remaining cases we have been able to reproduce the best cut in this benchmark.


After the tracking process of each frame, we perform region of interest (RoI) pooling #REFR on h t according to the predicted box to obtain the target region features which are further flattened to obtain the historical target patch embeddings.

More concretely, grounding is performed on a standalone image but tracking is executed on consecutive video which means there exist historical appearances of the Figure 3 . Architecture of the proposed semantic-guided temporal modeling module.

To improve the adaptability to the variations of the target using the temporal information, we propose a Semantics-Guided Temporal Modeling (SGTM) module.

It is designed to learn the temporal clue about recent target appearances from historical target states with semantics from the natural language as guidance.

Figure 3 shows the architecture of the SGTM module, which mainly consists of a transformer encoder and decoder.

Besides, we also take into account the global semantic representation h 1 l of the Natural Language (NL) outputted by the multi-source relation modeling module, which is called the NL CLS token denoted as h CLS l .

We directly concatenate the NL CLS token with the historical target patch embeddings and feed them into the encoder.

Herein, the encoder is used to enhance the token embeddings corresponding to the target region and suppress those corresponding to the noise region with the guidance of NL CLS.

After that, we utilize a decoder to compute cross-attention between enhanced target patch embeddings and a temporal query which is a learnable vector.

The temporal query can attend to the language context and historical target embeddings, thus learning a robust historical target representation for future tracking.

Joint Visual Grounding And Tracking With Natural Language Specification

Abstract: Tracking by natural language specification aims to locate the referred target in a sequence based on the natural language description. Existing algorithms solve this issue in two steps, visual grounding and tracking, and accordingly deploy the separated grounding model and tracking model to implement these two steps, respectively. Such a separated framework overlooks the link between visual grounding and tracking, which is that the natural language descriptions provide global semantic cues for localizing the target for both two steps. Besides, the separated framework can hardly be trained end-to-end. To handle these issues, we propose a joint visual grounding and tracking framework, which reformulates grounding and tracking as a unified task: localizing the referred target based on the given visual-language references. Specifically, we propose a multi-source relation modeling module to effectively build the relation between the visual-language references and the test image. In addition, we design a temporal modeling module to provide a temporal clue with the guidance of the global semantic information for our model, which effectively improves the adaptability to the appearance variations of the target. Extensive experimental results on TNL2K, LaSOT, OTB99, and RefCOCOg demonstrate that our method performs favorably against state-of-the-art algorithms for both tracking and grounding. Code is available at https://github.com/lizhou-cs/JointNLT.

Faster R-Cnn: Towards Real-Time Object Detection With Region Proposal Networks

Abstract: State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet [1] and Fast R-CNN [2] have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with ’attention’ mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model [3] , our detection system has a frame rate of 5 fps ( including all steps ) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available.


Before adapting the definition of the linearly implicit methods from #REFR to the PDE context at hand in 2.1.2, we recall some basic notations for classical Runge-Kutta methods applied to PDE problems and introduce their (linear) stability function in Section 2.1.1.

In this section, we introduce the linearly implicit methods from #OTHEREFR adapted to the PDE setting of the equation 1as well as the associated functional framework.

2.1. Description of the linearly implicit methods.

Linearly implicit method from #OTHEREFR applied to a semilinear PDE of the form (1) rely on a classical Runge-Kutta method, referred to as the "underlying Runger-Kutta method" in the following, as well as on additional variables that are explicitly updated at every time step.

2.1.1. The underlying Runge-Kutta method.

We fix a number of stages s P N ‹ and we consider throughout this paper a Runge-Kutta method with s P N ‹ stages, and real coefficients pa ij q 1ďi,jďs , pb i q 1ďiďs , pc i q 1ďiďs .

We group these coefficients in a square matrix A of size s with coefficients pa i,j q 1ďi,jďs , a column vector b of size s with coefficients pb i q 1ďiďs , and a column vector c of size s with coefficients pc i q 1ďiďs .

We moreover denote by 1 the column vector of size s with all components equal to 1.

We recall that the (linear) stability function of the method is defined for λ P C by #OTHEREFR Rpλq " 1`λb t pI´λAq´11,

High Order Linearly Implicit Methods For Semilinear Evolution Pdes

Abstract: This paper considers the numerical integration of semilinear evolution PDEs using the high order linearly implicit methods developped in [21] in the ODE setting. These methods use a collocation Runge-Kutta method as a basis, and additional variables that are updated explicitly and make the implicit part of the collocation Runge-Kutta method only linearly implicit. In this paper, we introduce several notions of stability for the underlying Runge-Kutta methods as well as for the explicit step on the additional variables necessary to fit the context of evolution PDE. We prove a main theorem about the high order of convergence of these linearly implicit methods in this PDE setting, using the stability hypotheses introduced before. We use nonlinear Schrödinger equations and heat equations as main examples but our results extend beyond these two classes of evolution PDEs. We illustrate our main result numerically in dimensions 1 and 2, and we compare the efficiency of the linearly implicit methods with other methods from the litterature. We also illustrate numerically the necessity of the stability conditions of our main result.

High Order Linearly Implicit Methods For Evolution Equations: How To Solve An Ode By Inverting Only Linear Systems

Abstract: This paper introduces a new class of numerical methods for the time integration of evolution equations set as Cauchy problems of ODEs or PDEs. The systematic design of these methods mixes the Runge-Kutta collocation formalism with collocation techniques, in such a way that the methods are linearly implicit and have high order. The fact that these methods are implicit allows to avoid CFL conditions when the large systems to integrate come from the space discretization of evolution PDEs. Moreover, these methods are expected to be efficient since they only require to solve one linear system of equations at each time step, and efficient techniques from the literature can be used to do so. After the introduction of the methods, we set suitable definitions of consistency and stability for these methods. This allows for a proof that arbitrarily high order linearly implicit methods exist and converge when applied to ODEs. Eventually, we perform numerical experiments on ODEs and PDEs that illustrate our theoretical results for ODEs, and compare our methods with standard methods for several evolution PDEs.

classical Runge-Kutta methods
linearly implicit methods


Let U and V denote the sets of benign users (clients) #REFR and items, respectively. |U| and |V | are users' and items' sizes.

Following #OTHEREFR , we employ the most commonly used federated recommendation framework proposed by #OTHEREFR as our base FedRec framework.

Most existing FedRecs #OTHEREFR are based on this framework.

In FedRec, each user is a client who manages its' local training dataset D .

D consists of many user-item interactions ( , , ), where is a binary rating denoting whether user has interacted with item .

i.e., = 1 means has interacted with , while = 0 indicates no interaction between and .

V + and V − are used to denote the sets of interacted items and non-interacted items for user .

The FedRec aims to predict^between and each non-interacted item and recommend the top-K ones with the highest predicted scores.

Manipulating Federated Recommender Systems: Poisoning With Synthetic Users And Its Countermeasures

Abstract: Federated Recommender Systems (FedRecs) are considered privacypreserving techniques to collaboratively learn a recommendation model without sharing user data. Since all participants can directly influence the systems by uploading gradients, FedRecs are vulnerable to poisoning attacks of malicious clients. However, most existing poisoning attacks on FedRecs are either based on some prior knowledge or with less effectiveness. To reveal the real vulnerability of FedRecs, in this paper, we present a new poisoning attack method to manipulate target items' ranks and exposure rates effectively in the top-recommendation without relying on any prior knowledge. Specifically, our attack manipulates target items' exposure rate by a group of synthetic malicious users who upload poisoned gradients considering target items' alternative products. We conduct extensive experiments with two widely used FedRecs (Fed-NCF and Fed-LightGCN) on two real-world recommendation datasets. The experimental results show that our attack can significantly improve the exposure rate of unpopular target items with extremely fewer malicious users and fewer global epochs than state-of-the-art attacks. In addition to disclosing the security hole, we design a novel countermeasure for poisoning attacks on FedRecs. Specifically, we propose a hierarchical gradient clipping with sparsified updating to defend against existing poisoning attacks. The empirical results demonstrate that the proposed defending mechanism improves the robustness of FedRecs. CCS CONCEPTS • Information systems → Recommender systems; • Security and privacy → Web application security.

Managing Popularity Bias In Recommender Systems With Personalized Re-Ranking

Abstract: Many recommender systems suffer from popularity bias: popular items are recommended frequently while less popular, niche products, are recommended rarely or not at all. However, recommending the ignored products in the `long tail' is critical for businesses as they are less likely to be discovered. In this paper, we introduce a personalized diversification re-ranking approach to increase the representation of less popular items in recommendations while maintaining acceptable recommendation accuracy. Our approach is a post-processing step that can be applied to the output of any recommender system. We show that our approach is capable of managing popularity bias more effectively, compared with an existing method based on regularization. We also examine both new and existing metrics to measure the coverage of long-tail items in the recommendation.


However, our work is the first to analyze the synergy between Mean Teacher #REFR and contrastive learning.

To align contrastive pre-training with downstream tasks other than image classification (e.g., object detection, semantic segmentation), more fine-grained approaches have been proposed based on masks #OTHEREFR , objects #OTHEREFR , or regions #OTHEREFR .

Our object-level contrastive learning strategy is inspired by this line of research.

Instead of applying contrastive learning in pre-training visual backbones, we study how to improve domain adaptive object detectors using noisy pseudo-labels and object-level contrast.

Technically, we construct contrastive pairs using the predicted classes in pseudo-labels and optimize multi-scale features, both of which are different from typical objectlevel contrastive learning.

Recently, contrastive learning is explored in teacher-student learning for detection #OTHEREFR .

Moreover, we present a simple and general framework CMT that does not rely on negative sample mining or selection.

Contrastive Mean Teacher For Domain Adaptive Object Detectors

Abstract: Object detectors often suffer from the domain gap between training (source domain) and real-world applications (target domain). Mean-teacher self-training is a powerful paradigm in unsupervised domain adaptation for object detection, but it struggles with low-quality pseudo-labels. In this work, we identify the intriguing alignment and synergy between mean-teacher self-training and contrastive learning. Motivated by this, we propose Contrastive Mean Teacher (CMT)-a unified, general-purpose framework with the two paradigms naturally integrated to maximize beneficial learning signals. Instead of using pseudo-labels solely for final predictions, our strategy extracts objectlevel features using pseudo-labels and optimizes them via contrastive learning, without requiring labels in the target domain. When combined with recent mean-teacher selftraining methods, CMT leads to new state-of-the-art targetdomain performance: 51.9% mAP on Foggy Cityscapes, outperforming the previously best by 2.1% mAP. Notably, CMT can stabilize performance and provide more significant gains as pseudo-label noise increases.

Weight-Averaged Consistency Targets Improve Semi-Supervised Deep Learning Results

Abstract: The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change only once per epoch, Temporal Ensembling becomes unwieldy when learning large datasets. To overcome this problem, we propose Mean Teacher, a method that averages model weights instead of label predictions. As an additional benefit, Mean Teacher improves test accuracy and enables training with fewer labels than Temporal Ensembling. Without changing the network architecture, Mean Teacher achieves an error rate of 4.35% on SVHN with 250 labels, outperforming Temporal Ensembling trained with 1000 labels. We also show that a good network architecture is crucial to performance. Combining Mean Teacher and Residual Networks, we improve the state of the art on CIFAR-10 with 4000 labels from 10.55% to 6.28%, and on ImageNet 2012 with 10% of the labels from 35.24% to 9.11%.


For the supplementary experiment aimed at discovering five target-hitting molecules (Supplementary Note 1), another fragment set containing 2,547 BRICS fragments that appeared more than 100 times in the training set of the MOSES database #REFR was used.

Two types of fragment sets were used in this study.

One set contained 2,207 BRICS fragments that appeared more than 150 times in the training set #OTHEREFR curated from the ChEMBL database #OTHEREFR (release version 25).

This fragment set was used for the three major experiments: the discovery of seven targethitting molecules, protein docking materials, and HIV inhibitors.

The detailed reasoning for using these fragment sets is described in Supplementary Note 2.

Materials Discovery With Extreme Properties Via Ai-Driven Combinatorial Chemistry

Abstract: The goal of most materials discovery is to discover materials that are superior to those currently known. Fundamentally, this is close to extrapolation, which is a weak point for most machine learning models that learn the probability distribution of data. Herein, we develop AIdriven combinatorial chemistry, which is a rule-based inverse molecular designer that does not rely on data. Since our model has the potential to generate all possible molecular structures that can be obtained from combinations of molecular fragments, unknown materials with superior properties can be discovered. We theoretically and empirically demonstrate that our model is more suitable for discovering better materials than probability distribution-learning models. In an experiment aimed at discovering molecules that hit seven target properties, our model discovered 1,315 of all target-hitting molecules and 7,629 of five target-hitting molecules out of 100,000 trials, whereas the probability distributionlearning models failed. To illustrate the performance in actual problems, 1

Molecular Sets (Moses): A Benchmarking Platform For Molecular Generation Models

Abstract: Deep generative models such as generative adversarial networks, variational autoencoders, and autoregressive models are rapidly growing in popularity for the discovery of new molecules and materials. In this work, we introduce MOlecular SEtS (MOSES), a benchmarking platform to support research on machine learning for drug discovery. MOSES implements several popular molecular generation models and includes a set of metrics that evaluate the diversity and quality of generated molecules. MOSES is meant to standardize the research on molecular generation and facilitate the sharing and comparison of new models. Additionally, we provide a large-scale comparison of existing state of the art models and elaborate on current challenges for generative models that might prove fertile ground for new research. Our platform and source code are freely available at this https URL.


Definition 8 (Valid Control Refinement):

The next theorem states that similar to the (ε, δ)approximate simulation relation defined in #REFR , there always exists at least one valid control refinement for our newly defined SSR. Proof.

upper-bounds the sub-probability coupling v, namelȳ

for all measurable sets A ⊂X × X, all (x k , x k ) ∈ R, and allû k ∈Û.

Note that for a model class M(θ) that is in relation with a model M the right-hand side of (12) can depend on θ and the left-hand side can depend on θ only via the dynamics of M(θ) represented by the kernel t, i.e.,

(x + ,x + )∈A s(dx + |x k , x k , x k+1 ,û k )t(dx + |x k , i(x k , x k ,û k ); θ) ≥ v(A|x k , x k ,û k ; θ).

If M δ ε M(θ), then this implies that there also exists a probabilistic kernelv(•|•; θ) satisfying the conditions for an (ε, δ)-approximate probabilistic simulation relation (cf. #OTHEREFR , Def. 9).

Furthermore, we can then compute the state mapping s as the conditional kernel obtained fromv(

•|•; θ), that is, s(dx k+1 |x k , x k , x k+1 ,û k ) :=v(dx k+1 |x k , x k , x k+1 ,û k ; θ),

where the right term is conditioned on x k+1 such that

v(dx k+1 × dx k+1 |x k , x k , x k+1 ,û k ; θ) =v(dx k+1 |x k , x k , x k+1 ,û k ; θ)t(dx k+1 |x k , i(•); θ).

Bayesian Approach To Temporal Logic Control Of Uncertain Systems

Abstract: This paper addresses the problem of data-driven computation of controllers that are correct by design for safetycritical systems and can provably satisfy (complex) functional requirements. With focus on continuous-state uncertain systems, we propose a two-stage approach that decomposes the problem into a learning stage and a robust formal controller synthesis stage. The first stage utilizes available Bayesian regression results to compute robust credible sets for the true parameters of the system. For the second stage, we introduce methods for systems subject to both stochastic and parametric uncertainties. We provide for the first time simulation relations for enabling correct-by-design control refinement that are founded on coupling uncertainties of stochastic systems via subprobability measures. The presented relations are essential for constructing abstract models that are related to not only one model but to a set of parameterized models. The results are demonstrated on a linear model and the nonlinear model of the Van der Pol Oscillator.

Verification Of General Markov Decision Processes By Approximate Similarity Relations And Policy Refinement

Abstract: In this work we introduce new approximate similarity relations that are shown to be key for policy (or control) synthesis over general Markov decision processes. The models of interest are discrete-time Markov decision processes, endowed with uncountably-infinite state spaces and metric output (or observation) spaces. The new relations, underpinned by the use of metrics, allow in particular for a useful trade-off between deviations over probability distributions on states, and distances between model outputs. We show that the new probabilistic similarity relations, inspired by a notion of simulation developed for finite-state models, can be effectively employed over general Markov decision processes for verification purposes, and specifically for control refinement from abstract models.


As a result of the branch dynamics of the complex system and the stochastic algorithm that is still undetermined, it is challenging to manage the collective dynamics #REFR .

To implement the evolutionary Lagrange technique, a method like Lin et al. (2003) integrated into the evolutionary min-max algorithm #OTHEREFR .

This EA combines self-adaptation for penalty parameters to achieve global integration, allowing for the use of smaller penalty parameters without impacting the final search results.

Besides, the Lagrange interpolating polynomial is the solitary polynomial of the lowest degree that interpolates a given collection of data in numerical analysis and data mining.

For mathematicians, the foundations of mathematics are a dynamic complex system.

A method for solving discrete optimization problems relied on the interpretation of calculus algorithms.

EA seems to be a useful tool for optimizing complex and non-convex optimization problems.

The earlier algorithms, such as Suma (2021), employing EA-based network reconstruction methods in the proposed new algorithm #OTHEREFR , have concentrated on the broad applicability of EA-based methodologies and their enhanced efficiency in solving reconstruction situations in complicated applications.

Also, both the objective and constraint functions are non-convex and incorporate expectations over random states to find a stationary point for a broad non-convex stochastic optimization problem. In addition, Liu et al.

(2019) presented a constrained stochastic successive convex approximation algorithm #OTHEREFR .

In several deliberations, it was demonstrated that using a hybrid algorithm and Lagrange technique to solve limited optimization problems.

Leo: Lagrange Elementary Optimization

Abstract: Global optimization problems are frequently solved using the practical and efficient method of evolutionary sophistication. But as the original problem becomes more complex, so does its efficacy and expandability. Thus, the purpose of this research is to introduce the Lagrange Elementary Optimization (Leo) as an evolutionary method, which is self-adaptive inspired by the remarkable accuracy of vaccinations using the albumin quotient of human blood. They develop intelligent agents using their fitness function value after gene crossing. These genes direct the search agents during both exploration and exploitation. The main objective of the Leo algorithm is presented in this paper along with the inspiration and motivation for the concept. To demonstrate its precision, the proposed algorithm is validated against a variety of test functions, including 19 traditional benchmark functions and the CECC06 2019 test functions. The results of Leo for 19 classic benchmark test functions are evaluated against DA, PSO, and GA separately, and then two other recent algorithms such as FDO and LPB are also included in the evaluation. In addition, the Leo is tested by ten functions on CECC06 2019 with DA, WOA, SSA, FDO, LPB, and FOX algorithms distinctly. The cumulative outcomes demonstrate Leo's capacity to increase the starting population and move toward the global optimum. Different standard measurements are used to verify and prove the stability of Leo in both the exploration and exploitation phases. Moreover, Statistical analysis supports the findings results of the proposed research. Finally, novel applications in the real world are introduced to demonstrate the practicality of Leo.

Dr$^{2}$-Net: Deep Residual Reconstruction Network For Image Compressive Sensing

Abstract: Most traditional algorithms for compressive sensing image reconstruction suffer from the intensive computation. Recently, deep learning-based reconstruction algorithms have been reported, which dramatically reduce the time complexity than iterative reconstruction algorithms. In this paper, we propose a novel Deep Residual Reconstruction Network (DR2-Net) to reconstruct the image from its Compressively Sensed (CS) measurement. The DR2-Net is proposed based on two observations: (1) linear mapping could reconstruct a high-quality preliminary image, and (2) residual learning could further improve the reconstruction quality. Accordingly, DR2-Net consists of two components, i.e., linear mapping network and residual network, respectively. Specifically, the fully-connected layer in neural network implements the linear mapping network. We then expand the linear mapping network to DR2-Net by adding several residual learning blocks to enhance the preliminary image. Extensive experiments demonstrate that the DR2-Net outperforms traditional iterative methods and recent deep learning-based methods by large margins at measurement rates 0.01, 0.04, 0.1, and 0.25, respectively. The code of DR2-Net has been released on: https://github.com/coldrainyht/caffe_dr2 .


To address this issue, Neural Radiance Fields (NeRFs) #REFR directly optimizes a continuous radiance field representation using a multi-layer perceptron.

Traditional methods use Structure-From-Motion #OTHEREFR and bundle adjustment #OTHEREFR to reconstruct colored point clouds.

More recently, a number of learning-based novel view synthesis methods have been proposed, but they require 3D geometry as inputs #OTHEREFR .

Recently, neural volume rendering approaches have been used to reconstruct scenes from only images #OTHEREFR .

However, the rendering resolution of these methods is limited by the time and computational complexity of the discretely sampled volumes.

This allows for synthesizing novel views of realistic images at an unprecedented level of fidelity.

Various extensions of NeRF have also been proposed to improve efficiency #OTHEREFR Yu et al., 2021; #OTHEREFR , generalization #OTHEREFR Yu et al., 2022; #OTHEREFR , quality #OTHEREFR , compositionality #OTHEREFR etc.

While these neural methods produce high-quality novel views of a scene, they do not support relighting. In contrast, our approach aims at relightable appearance reconstruction.

Learning-based methods for relighting without explicit geometry have been proposed #OTHEREFR , but they are not 3D-aware.

Learning Object-Centric Neural Scattering Functions For Free-Viewpoint Relighting And Scene Composition

Abstract: Photorealistic object appearance modeling from 2D images is a constant topic in vision and graphics. While neural implicit methods (such as Neural Radiance Fields) have shown high-fidelity view synthesis results, they cannot relight the captured objects. More recent neural inverse rendering approaches have enabled object relighting, but they represent surface properties as simple BRDFs, and therefore cannot handle translucent objects. We propose Object-Centric Neural Scattering Functions (OSFs) for learning to reconstruct object appearance from only images. OSFs not only support free-viewpoint object relighting, but also can model both opaque and translucent objects. While accurately modeling subsurface light transport for translucent objects can be highly complex and even intractable for neural methods, OSFs learn to approximate the radiance transfer from a distant light to an outgoing direction at any spatial location. This approximation avoids explicitly modeling complex subsurface scattering, making learning a neural implicit model tractable. Experiments on real and synthetic data show that OSFs accurately reconstruct appearances for both opaque and translucent objects, allowing faithful free-viewpoint relighting as well as scene composition. In our supplementary material, we include a video for an overview.

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.


B. More Implementation Details B.1. Model Architecture And Training Details

Since we the Kinetics pre-trained models adopt sparse sampling #REFR , we use a frame span of 300 for action detection, which is the default frame number of Kinetics videos. More details are listed in Table 23 .

We adopt the Stage-1 pre-trained video encoder and add an extra classification layer for finetuning.

Detailed hyperparameters for different datasets are shown in Table 22 .

In our experiments, we have tried to fine-tune the Stage-2 pre-trained video encoder, but the results on Kinetics are similar.

Following VideoMAE #OTHEREFR and ST-MAE #OTHEREFR , we add ROIAlign with MaxPooling to generate the regions of interest.

For fine-tuning, we adopt the same architecture as in Stage 2, but we only apply VTC and VTM losses.

For all datasets, we sparsely sample 12 frames for both training and testing. More details are listed in Table 26 .

For a fair comparison, we follow Singularity #OTHEREFR to apply flip augmentation for SSV2 retrieval, which may harm the performance of this temporal-related dataset. Video question-answering.

Following the previous works #OTHEREFR , we formulate this task as text generation instead of classification.

Unmasked Teacher: Towards Training-Efficient Video Foundation Models

Abstract: Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain. Although VideoMAE has trained a robust ViT from limited data, its low-level reconstruction poses convergence difficulties and conflicts with high-level cross-modal alignment. This paper proposes a training-efficient method for temporal-sensitive VFMs that integrates the benefits of existing methods. To increase data efficiency, we mask out most of the low-semantics video tokens, but selectively align the unmasked tokens with IFM, which serves as the UnMasked Teacher (UMT). By providing semantic guidance, our method enables faster convergence and multimodal friendliness. With a progressive pre-training framework, our model can handle various tasks including scenerelated, temporal-related, and complex video-language understanding. Using only public sources for pre-training in 6 days on 32 A100 GPUs, our scratch-built ViT-L/16 achieves state-of-the-art performances on various video tasks. The code and models will be released at https://github. com/OpenGVLab/unmasked_teacher.

Temporal Segment Networks: Towards Good Practices For Deep Action Recognition

Abstract: Deep convolutional networks have achieved great success for visual recognition in still images. However, for action recognition in videos, the advantage over traditional methods is not so evident. This paper aims to discover the principles to design effective ConvNet architectures for action recognition in videos and learn these models given limited training samples. Our first contribution is temporal segment network (TSN), a novel framework for video-based action recognition. which is based on the idea of long-range temporal structure modeling. It combines a sparse temporal sampling strategy and video-level supervision to enable efficient and effective learning using the whole action video. The other contribution is our study on a series of good practices in learning ConvNets on video data with the help of temporal segment network. Our approach obtains the state-the-of-art performance on the datasets of HMDB51 (\( 69.4\,\% \)) and UCF101 (\( 94.2\,\% \)). We also visualize the learned ConvNet models, which qualitatively demonstrates the effectiveness of temporal segment network and the proposed good practices (Models and code at https://github.com/yjxiong/temporal-segment-networks).


Theorem 1 is consistent with the result in #REFR . given by C = {000, 112, 220, 332}.

Z i = ⌊ m 2 ⌋ j=1 y x j u∈Ci∪C ′ i π(i) a=1 v∈Eija χ(u i • v i ) = ⌊ m 2 ⌋ j=1 y x j u∈Ci∪C ′ i π(i) a=1 v i ∈V i ,w L (v ia )=j; w L (v i b )<j for b<a;w L (v i b )≤j for b>a χ(u i • v i ) = 2 ⌊ m 2 ⌋ j=1 y x j u∈Ci∪C ′ i π(i) a=1 cos 2πu ia j m b<a 1 + 2 j−1 t=1 cos 2πu i b t m b>a 1 + 2 j t=1 cos 2πu i b t m = 2 ⌊ m 2 ⌋ j=1 y x j |C i | (2j + 1) π(i) − (2j − 1) π(i) 2 + u∈C ′ i π(i) a=1 cos 2πu ia j m b<a 1 + 2 j−1 t=1 cos 2πu i b t m b>a 1 + 2 j t=1 cos 2πu i b t m .

The result then follows from (3.1) and (32).

The case for m even can be proved in a similar way.

Note that when we consider the pomset block metric over Z s 2 and Z s 3 , the pomset block metric will coincide with poset block metric.

Macwilliams Type Identities For Linear Block Codes On Certain Pomsets

Abstract: Pomset block metric is a generalization of pomset metric. In this paper, we define weight enumerator of linear block codes in pomset metric over Z m and establish MacWilliams type identities for linear block codes with respect to certain pomsets. The relation between weight enumerators of two linear pomset block codes and their direct sum is also investigated.

Classification Of Poset-Block Spaces Admitting Macwilliams-Type Identity

Abstract: In this work we prove that a poset-block space admits a MacWilliams-type identity if and only if the poset is hierarchical and at any level of the poset, all the blocks have the same dimension. When the poset-block admits the MacWilliams-type identity we explicit the relation between the weight enumerators of a code and its dual.


The SPICE #REFR metric on COCO caption is used to evaluate the VPG's visual perception ability, whereas SPICE is mainly designed for checking the objects, attributes, and relationships in the generated captions. As shown in Fig.

We notice an interesting phenomenon that when transferring to a given LLM tgt , both the convergence rate and optimal performance are roughly inversely proportional to the size of LLM src .

For example, as shown in Table 3 , the OPT 125M→2.7B and OPT 350M→2.7B have much higher speed-up rate than OPT 1.3B→2.7B on all of the datasets. Meanwhile, as demonstrated in Fig.

7 , the optimal performance of OPT 125M→2.7B and OPT 350M→2.7B are better than OPT 1.3B→2.7B on 3 VQA tasks.

We hypothesize that training VPG on larger LLM will have a worse influence on VPG's existing finegrained perception ability, due to the enlarging embedding dimensions.

To validate our hypothesis, we fix the VPG weight and only tune linear projectors to test VPGs trained with different LLM src through cross-size transfer.

8 , for each row, given the LLM tar , the performance of VPG trained on smaller LLM src can outperform the larger ones for most conditions, which indicates a better visual perception ability of VPG trained on smaller LLM src .

Therefore, adapting a VPG from a smaller LLM which is less affected, is helpful to take fewer steps to reach the TFS's best performance and achieve even better performance.

As we illustrated before, FlanT5 large training is extremely unstable.

Even when the learning rate is adjusted to one-tenth of its original value, the model does not converge or shows a very slow convergence rate after 4 epochs.

However, we find that by lowering the learning rate for stage-2 training, our VPGTrans can achieve stable training on FlanT5 large .

Transfer Visual Prompt Generator Across Llms

Abstract: While developing a new vision-language LLM (VL-LLM) by pre-training on tremendous image-text pairs from scratch can be exceedingly resource-consuming, connecting an existing LLM with a comparatively lightweight visual prompt generator (VPG) becomes a feasible paradigm. However, further tuning the VPG part of the VL-LLM still suffers from indispensable computational costs, i.e., requiring thousands of GPU hours and millions of training data. One alternative solution is to transfer an existing VPG from any existing VL-LLMs for the target VL-LLM.

Spice: Semantic Propositional Image Caption Evaluation

Abstract: There is considerable interest in the task of automatically generating image captions. However, evaluation is challenging. Existing automatic evaluation metrics are primarily sensitive to n-gram overlap, which is neither necessary nor sufficient for the task of simulating human judgment. We hypothesize that semantic propositional content is an important component of human caption evaluation, and propose a new automated caption evaluation metric defined over scene graphs coined SPICE. Extensive evaluations across a range of models and datasets indicate that SPICE captures human judgments over model-generated captions better than other automatic metrics (e.g., system-level correlation of 0.88 with human judgments on the MS COCO dataset, versus 0.43 for CIDEr and 0.53 for METEOR). Furthermore, SPICE can answer questions such as which caption-generator best understands colors? and can caption-generators count?

VPG's visual perception
generated captions


The text data is tokenized using a SentencePiece #REFR tokenizer with a vocabulary size of 64k.

We randomly initialize the model parameters using the method described in BEiT #OTHEREFR .

The image resolution is set to 224×224, and the patch size is 16×16. The maximum sequence length for text is 128.

We use a batch size of 6, 144 and train the model from scratch for 200k steps, which is equivalent to 40 epochs of the image-text pairs.

Each batch contains 2, 048 images, 2, 048 texts, and 2, 048 image-text pairs.

We perform image augmentation using random resized cropping, horizontal flipping, and color jittering, following the same method as BEiT #OTHEREFR .

We use the Adam optimizer #OTHEREFR with β 1 = 0.9 and β 2 = 0.999 to optimize the model.

The peak learning rate is 2e-3, and we use linear warmup for the first 10, 000 steps and cosine learning rate decay.

The weight decay is 0.05, and we disable dropout and use stochastic depth #OTHEREFR with a rate of 0.1.

The three pretrain losses are equally weighted as in BEIT-3 #OTHEREFR .

Scaling Vision-Language Models With Sparse Mixture Of Experts

Abstract: The field of natural language processing (NLP) has made significant strides in recent years, particularly in the development of large-scale vision-language models (VLMs). These models aim to bridge the gap between text and visual information, enabling a more comprehensive understanding of multimedia data. However, as these models become larger and more complex, they also become more challenging to train and deploy. One approach to addressing this challenge is the use of sparsely-gated mixture-of-experts (MoE) techniques, which divide the model into smaller, specialized sub-models that can jointly solve a task. In this paper, we explore the effectiveness of MoE in scaling vision-language models, demonstrating its potential to achieve state-of-theart performance on a range of benchmarks over dense models of equivalent computational cost. Our research offers valuable insights into stabilizing the training of MoE models, understanding the impact of MoE on model interpretability, and balancing the trade-offs between compute performance when scaling VLMs. We hope our work will inspire further research into the use of MoE for scaling large-scale visionlanguage models and other multimodal machine learning applications.

Sentencepiece: A Simple And Language Independent Subword Tokenizer And Detokenizer For Neural Text Processing

Abstract: This paper describes SentencePiece, a language-independent subword tokenizer and detokenizer designed for Neural-based text processing, including Neural Machine Translation. It provides open-source C++ and Python implementations for subword units. While existing subword segmentation tools assume that the input is pre-tokenized into word sequences, SentencePiece can train subword models directly from raw sentences, which allows us to make a purely end-to-end and language independent system. We perform a validation experiment of NMT on English-Japanese machine translation, and find that it is possible to achieve comparable accuracy to direct subword training from raw sentences. We also compare the performance of subword training and segmentation with various configurations. SentencePiece is available under the Apache 2 license at https://github.com/google/sentencepiece.