A. Decomposed Reward Functions & Primitive Actions

Further, following #REFR , each action a is parameterized as a motion primitive ζ that is executed at a 3D position projected by a pixel P of an orthographic RGB-D heightmap of the scene, i.e., a = ( ζ, P ) .

Under the property-guided Q-Map learning framework, individual sub-rewards (components) r k account for properties that are semantic (high-level) and task-relevant.

As an example, criteria for a flying agent to land on appropriate locations could consist of 1) being on flat surfaces and 2) being away from crowds.

Each criterion can be represented as a sub-reward which serves as an indicator function I(C), denoting whether a task-specific property (C) was identified.

This approach enables the use of property-linked rewards rather than low-level feature rewards, such as rewarding the agent for being in close proximity of targets, where the action is the low-level visuomotor control.

Although sub-rewards are created manually based on task criteria, we argue that the decomposition of the overall task reward could be automated via extensions to approaches like #OTHEREFR , which extracts reward channels from agent interaction data, and we defer this to future work.

Additionally, learning property representations is similar to discovering concepts in images #OTHEREFR .

A Closer Look At Reward Decomposition For High-Level Robotic Explanations

Abstract: Explaining the behavior of intelligent agents such as robots to humans is challenging due to their incomprehensible proprioceptive states, variational intermediate goals, and resultant unpredictability. Moreover, one-step explanations for reinforcement learning agents can be ambiguous as they fail to account for the agent's future behavior at each transition, adding to the complexity of explaining robot actions. By leveraging abstracted actions that map to task-specific primitives, we avoid explanations on the movement level. Our proposed framework combines reward decomposition (RD) with abstracted action spaces into an explainable learning framework, allowing for non-ambiguous and high-level explanations based on object properties in the task. We demonstrate the effectiveness of our framework through quantitative and qualitative analysis of two robot scenarios, showcasing visual and textual explanations, from output artifacts of RD explanation, that are easy for humans to comprehend. Additionally, we demonstrate the versatility of integrating these artifacts with large language models for reasoning and interactive querying.

Learning Synergies Between Pushing And Grasping With Self-Supervised Deep Reinforcement Learning

Abstract: Skilled robotic manipulation benefits from complex synergies between non-prehensile (e.g. pushing) and prehensile (e.g. grasping) actions: pushing can help rearrange cluttered objects to make space for arms and fingers; likewise, grasping can help displace objects to make pushing movements more precise and collision-free. In this work, we demonstrate that it is possible to discover and learn these synergies from scratch through model-free deep reinforcement learning. Our method involves training two fully convolutional networks that map from visual observations to actions: one infers the utility of pushes for a dense pixel-wise sampling of end effector orientations and locations, while the other does the same for grasping. Both networks are trained jointly in a Q-learning framework and are entirely self-supervised by trial and error, where rewards are provided from successful grasps. In this way, our policy learns pushing motions that enable future grasps, while learning grasps that can leverage past pushes. During picking experiments in both simulation and real-world scenarios, we find that our system quickly learns complex behaviors amid challenging cases of clutter, and achieves better grasping success rates and picking efficiencies than baseline alternatives after only a few hours of training. We further demonstrate that our method is capable of generalizing to novel objects. Qualitative results (videos), code, pre-trained models, and simulation environments are available at http://vpg.cs.princeton.edu


Stacked Conditional GAN #REFR is another technique for shadow detection that uses GAN. Nevertheless, this approach needs shadow-free photos to work.

For the SBU dataset, their method produced good results, although the identification process was computationally expensive during testing. .

For shadow detection, our suggested method likewise makes use of adversarial training, but it differs significantly from scGAN in key ways.

While our approach utilizes a generator to produce enhanced training images in the RGB space, scGAN applies the generator to generate a binary shadow mask that is conditioned on the input image.

In contrast, scGAN employs the discriminator as a regulatory mechanism to encourage global consistency, it is more prominently used in our method to classify shadow pixels.

Unlike scGAN, our approach achieves real-time shadow detection without requiring output averaging or postprocessing steps.

A different recent method suggests using contextual information to improve shadow recognition.

By leveraging various spatial-directional RNN , our method incorporates both local and global information to enhance shadow detection performance on standard benchmarks.

Nonetheless, to further refine the results, a post-processing step utilizing a Conditional Random Field (CRF) is required.

In accordance with current trends in data augmentation, we provide a technique to enhance shadow detection using enhanced training samples. Zhang et al.

#OTHEREFR proposed a simple data augmentation strategy that enhances the network's generalization and resilience to adversarial examples by augmenting the dataset with linear combinations of pairs of examples and their corresponding labels. Shrivastava et al.

Ai Driven Shadow Model Detection In Agropv Farms

Abstract: Agro-photovoltaic (APV) is a burgeoning farming practice that combines agricultural activities and solar photovoltaic projects (PV) within the same area. This market is projected to experience significant growth in the next few years, with investments of approximately 9 billion US dollars expected in 2030, accounting for over 5% of the total photovoltaic investment. In certain countries, APV may even become the primary market for photovoltaic. The identification of shadows is a crucial aspect of APV research because shadows significantly impact several phenomena in the APV environment, including plant growth, microclimate, and evapotranspiration. Shadow detection can be highly beneficial, enabling a better understanding of the environment and ensuring a fair distribution of solar energy between PV and agricultural activities. In this study, we focus on using state-of-the-art CNN and GAN-based neural networks to identify shadows in agro-PV farms. CNN-based techniques can categorize pixels as either shadow or non-shadow by extracting features from input images using convolutional layers, while GANbased networks can produce synthetic shadow images and differentiate between fabricated and genuine images. Our case studies demonstrate the effectiveness of these techniques in detecting shadows in agro-PV farms, but we also acknowledge the challenges that need to be addressed, such as the impact of partial shadowing from moving objects like animals or people, as well as the need for real-time monitoring. We suggest that future research should focus on developing more sophisticated neural network-based shadow detection algorithms and integrating shadow detection with other control systems for agro-PV farms. Overall, this study emphasizes how important shadow detection is for increasing the productivity and profitability of agro-PV farms while offering environmental and social benefits that support the plants, soil, and farmers.

Stacked Conditional Generative Adversarial Networks For Jointly Learning Shadow Detection And Shadow Removal

Abstract: Understanding shadows from a single image spontaneously derives into two types of task in previous studies, containing shadow detection and shadow removal. In this paper, we present a multi-task perspective, which is not embraced by any existing work, to jointly learn both detection and removal in an end-to-end fashion that aims at enjoying the mutually improved benefits from each other. Our framework is based on a novel STacked Conditional Generative Adversarial Network (ST-CGAN), which is composed of two stacked CGANs, each with a generator and a discriminator. Specifically, a shadow image is fed into the first generator which produces a shadow detection mask. That shadow image, concatenated with its predicted mask, goes through the second generator in order to recover its shadow-free image consequently. In addition, the two corresponding discriminators are very likely to model higher level relationships and global scene characteristics for the detected shadow region and reconstruction via removing shadows, respectively. More importantly, for multi-task learning, our design of stacked paradigm provides a novel view which is notably different from the commonly used one as the multi-branch version. To fully evaluate the performance of our proposed framework, we construct the first large-scale benchmark with 1870 image triplets (shadow image, shadow mask image, and shadow-free image) under 135 scenes. Extensive experimental results consistently show the advantages of ST-CGAN over several representative state-of-the-art methods on two large-scale publicly available datasets and our newly released one.

shadow detection
Stacked Conditional GAN


In order to capture conceptually diverse embedding and exploiting the co-occurrence among objects, predicates and parents, we follow the technique used in BERT #REFR and mask randomly words from the input that must be predicted by the object/relation predictor.

Instead of using directly the sequence S as input in the Object/Relation Predictor, we embed s 1:T into e P 1:T , which allows to take into consideration different features: word embedding e w 1:T that allows to identify the class of the object (e.g. "button" or "container") or the relation (e.g.

"inside" or "right), object ID embedding e o 1:T to differentiate instances of the same object, relationship ID embedding e s 1:T to separate each relationship, type of word embedding e t 1:T to identify parts of a relationship (subject = 1, predicate = 2, object = 3) for example the part of the the relationship "button inside container" are "subject predicate object" and instead of writting the plain text, we rather associate an ID to each part of the relationship to differentiate them, and parent ID embedding e p 1:T , which is a feature that allows for each of the component to know its parent.

Those features are concatenated to form the input embedding for the object/relation predictor.

It is given by The object/relation predictor learns to produce three different outputs: (1) the contextualized feature vectors f 1:T , (2) the size vectors s 1:T , and (3) the position vectors p 1:T .

The contextualized feature vectors f 1:T describe objects, relations and their context with features from the input.

The size vectors s 1:T are predictions of the bounding boxes size for each object made by the object/relation predictor.

It is used as indicator later in the GUI layout generator to generate final bounding boxes size.

The position vectors p 1:T are predictions of bounding boxes position for each object.

Finally, in order to compute the objective function to train this part of the network, we need to predictê P 1:T from the features f 1:T using a single linear layer.

Indeed, since there is not ground truth for f 1:T , we predictê P 1:T from f 1:T and match it with e P 1:T to minimize the reconstruction error.

Guilget: Gui Layout Generation With Transformer

Abstract: Sketching out Graphical User Interface (GUI) layout is part of the pipeline of designing a GUI and a crucial task for the success of a software application. Arranging all components inside a GUI layout manually is a time-consuming task. In order to assist designers, we developed a method named GUILGET to automatically generate GUI layouts from positional constraints represented as GUI arrangement graphs (GUI-AGs). The goal is to support the initial step of GUI design by producing realistic and diverse GUI layouts. The existing image layout generation techniques often cannot incorporate GUI design constraints. Thus, GUILGET needs to adapt existing techniques to generate GUI layouts that obey to constraints specific to GUI designs. GUILGET is based on transformers in order to capture the semantic in relationships between elements from GUI-AG. Moreover, the model learns constraints through the minimization of losses responsible for placing each component inside its parent layout, for not letting components overlap if they are inside the same parent, and for component alignment. Our experiments, which are conducted on the CLAY dataset, reveal that our model has the best understanding of relationships from GUI-AG and has the best performances in most of evaluation metrics. Therefore, our work contributes to improved GUI layout generation by proposing a novel method that effectively accounts for the constraints on GUI elements and paves the road for a more efficient GUI design pipeline.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


As objects only need to be localized when they are visible, occlusions can be handled by simple re-association, but it has been shown that maintaining a hypothesis about the location of invisible objects can help reduce the number of identity switches #REFR .

Notably, most prior datasets and methods focus on localizing occluded objects with a bounding box.

In contrast, we focus on a more precise video object segmentation setting.

Moreover, rather than attempting to perfectly localize the invisible instance, which is not always possible in practice, we extend the setting of the problem to segmenting the occluder instead in ambiguous scenarios.

Finally, we introduce a clear distinction between containment and occlusion at the output level.

Object permanence methods in computer vision were mostly studied in the context of multi-object tracking -the task of localizing all the objects from a pre-defined vocabulary with bounding boxes and associating them over time based on identity #OTHEREFR .

To this end, most approaches rely on a simple constant velocity heuristic #OTHEREFR , which propagates the last observed location of an object with a linear motion model.

It is, however, only robust when the camera is static and the object velocity does not change significantly during the occlusion (e.g. because it is short).

More complex, heuristicbased methods include #OTHEREFR , which localize invisible objects by modeling inter-occlusion relationships, and #OTHEREFR which capitalizes on the correlation between the motion of visible and invisible instances.

More recently, several learning-based methods for localizing invisible objects have been proposed.

#OTHEREFR takes precomputed bounded boxes for visible objects as input and passes them through a recurrent network #OTHEREFR that is trained to predict the bounding box for the occluded target.

Tracking Through Containers And Occluders In The Wild

Abstract: Tracking objects with persistence in cluttered and dynamic environments remains a difficult challenge for computer vision systems. In this paper, we introduce TCOW, a new benchmark and model for visual tracking through heavy occlusion and containment. We set up a task where the goal is to, given a video sequence, segment both the projected extent of the target object, as well as the surrounding container or occluder whenever one exists. To study this task, we create a mixture of synthetic and annotated real datasets to support both supervised learning and structured evaluation of model performance under various forms of task variation, such as moving or nested containment. We evaluate two recent transformer-based video models and find that while they can be surprisingly capable of tracking targets under certain settings of task variation, there remains a considerable performance gap before we can claim a tracking model to have acquired a true notion of object permanence.

Simple Online And Realtime Tracking

Abstract: This paper explores a pragmatic approach to multiple object tracking where the main focus is to associate objects efficiently for online and realtime applications. To this end, detection quality is identified as a key factor influencing tracking performance, where changing the detector can improve tracking by up to 18.9%. Despite only using a rudimentary combination of familiar techniques such as the Kalman Filter and Hungarian algorithm for the tracking components, this approach achieves an accuracy comparable to state-of-the-art online trackers. Furthermore, due to the simplicity of our tracking method, the tracker updates at a rate of 260 Hz which is over 20x faster than other state-of-the-art trackers.


First, the constructions of #REFR utilize a public random seed for a collisionresistant hash function, so their crLDC definition is quantified over the randomness of the seed generation algorithm.

We require that for all adversaries PPT adversaries A there exists a negligible function ε L (•) such that for all λ ∈ N and all x ∈ Σ k 1 , we have Pr[Limit(A(y), ρ, δ, x, y, λ) = 1] ⩽ ε L (λ), where the probability is taken over A and y = Enc λ (x).

If dist is the normalized Hamming distance HAM, we say the code is a Hamming crLDC; if dist is the normalized edit distance ED, we say the code is a InsDel crLDC.

Here, ℓ is the locality, ρ is the error-rate, p is the success probability, and a function is negligible if it is o(x −c ) for all constants c > 0.

If q 2 = 2, we say that C is a family of binary crLDCs, and if q 1 = q 2 we simply write C λ [K, k, q 1 ].

Definition 1 closely follows the crLDC definition of Blocki et al. #OTHEREFR with a few modifications.

Our constructions do not require a public random seed so we omit this algorithm from our definition and instead quantify the security of our crLDC over a code family {C λ } λ∈N .

This quantification also captures the notion of asymptotic security when interacting with PPT adversaries, which differs from standard (r)LDC definitions that consider information-theoretic adversaries.

Moreover, #OTHEREFR requires the public random seed to be generated in an honest (i.e., trusted) way, and our definition and constructions circumvent this requirement.

Second, we slightly strengthen the security definition by tweaking the predicate Fool: in Definition 1, the adversary wins if there exists an index i (not necessarily known by the adversary) such that the probability the decoder outputs correctly on input i is less than p.

In contrast, #OTHEREFR requires the adversary to output corrupt codeword y ′ and a target index i such that the probability the decoder outputs correctly on index i is less than p.

Computationally Relaxed Locally Decodable Codes, Revisited *

Abstract: We revisit computationally relaxed locally decodable codes (crLDCs) (Blocki et al., Trans. Inf. Theory '21) and give two new constructions. Our first construction is a Hamming crLDC that is conceptually simpler than prior constructions, leveraging digital signature schemes and an appropriately chosen Hamming code. Our second construction is an extension of our Hamming crLDC to handle insertion-deletion (InsDel) errors, yielding an InsDel crLDC. This extension crucially relies on the noisy binary search techniques of Block et al. (FSTTCS '20) to handle InsDel errors. Both crLDC constructions have binary codeword alphabets, are resilient to a constant fraction of Hamming and InsDel errors, respectively, and under suitable parameter choices have poly-logarithmic locality and encoding length linear in the message length and polynomial in the security parameter. These parameters compare favorably to prior constructions in the poly-logarithmic locality regime.

Relaxed Locally Correctable Codes In Computationally Bounded Channels

Abstract: Error-correcting codes that admit local decoding and correcting algorithms have been the focus of much recent research due to their numerous theoretical and practical applications. The goal is to obtain the best possible tradeoffs between the number of queries the algorithm makes to its oracle (the locality of the task), and the amount of redundancy in the encoding (the information rate). ::: In Hamming's classical adversarial channel model, the current tradeoffs are dramatic, allowing either small locality, but superpolynomial blocklength, or small blocklength, but high locality. However, in the computationally bounded, adversarial channel model, proposed by Lipton (STACS 1994), constructions of locally decodable codes suddenly exhibit small locality and small blocklength. The first such constructions are due to Ostrovsky, Pandey and Sahai (ICALP 2007) who build private locally decodable codes under the assumption that one-way functions exist, and in the setting where the sender and receiver share a private key. ::: We study variants of locally decodable and locally correctable codes in computationally bounded, adversarial channels, under the much weaker assumption that collision-resistant hash functions exist, and with no public-key or private-key cryptographic setup. Specifically, we provide constructions of relaxed locally correctable and relaxed locally decodable codes over the binary alphabet, with constant information rate, and poly-logarithmic locality. Our constructions compare favorably with existing schemes built under much stronger cryptographic assumptions, and with their classical analogues in the computationally unbounded, Hamming channel. ::: Our constructions crucially employ collision-resistant hash functions and local expander graphs, extending ideas from recent cryptographic constructions of memory-hard functions.


Besides, the statistical channel inversion power control #REFR is implemented, i.e., the transmit power for each user is chosen such that = , where is the target received signal strength at the BS.

Since the number of users can be much larger than the pilot length, assigning orthogonal pilots to all the users is impractical.

Therefore, a unique pilot sequence √ p is allocated to each user, where

Despite being non-orthogonal, the set of pilot sequences achieves asymptotic orthogonality when → ∞ #OTHEREFR .

Then, the received pilot signal Y ( ) ∈ C × at the BS in the -th transmission block can be expressed as follows:

where A short data packet with payload bits needs to be transmitted for each active user, which is encoded as a codeword of symbols, denoted as s .

We further assume each symbol in a codeword has unit power.

Therefore, the received data signal at the BS in the -th transmission block Y ( ) ∈ C × can be expressed as follows:

Grant-Free Massive Random Access With Retransmission: Receiver Optimization And Performance Analysis

Abstract: There is an increasing demand of massive machine-type communication (mMTC) to provide scalable access for a large number of devices, which has prompted extensive investigation on grant-free massive random access (RA) in 5G and beyond wireless networks. Although many efficient signal processing algorithms have been developed, the limited radio resource for pilot transmission in grant-free massive RA systems makes accurate user activity detection and channel estimation challenging, which thereby compromises the communication reliability. In this paper, we adopt retransmission as a means to improve the quality of service (QoS) for grant-free massive RA. Specifically, by jointly leveraging the user activity correlation between adjacent transmission blocks and the historical channel estimation results, we first develop an activity-correlation-aware receiver for grant-free massive RA systems with retransmission based on the correlated approximate message passing (AMP) algorithm. Then, we analyze the performance of the proposed receiver, including the user activity detection, channel estimation, and data error, by resorting to the state evolution of the correlated AMP algorithm and the random matrix theory (RMT). Our analysis admits a tight closed-form approximation for frame error rate (FER) evaluation. Simulation results corroborate our theoretical analysis and demonstrate the effectiveness of X. Bian and J. Zhang are with the Department of Electronic and Computer Engineering,

Grant-Free Massive Mtc-Enabled Massive Mimo: A Compressive Sensing Approach

Abstract: A key challenge of massive MTC (mMTC), is the joint detection of device activity and decoding of data. The sparse characteristics of mMTC makes compressed sensing (CS) approaches a promising solution to the device detection problem. However, utilizing CS-based approaches for device detection along with channel estimation, and using the acquired estimates for coherent data transmission is suboptimal, especially when the goal is to convey only a few bits of data. First, we focus on the coherent transmission and demonstrate that it is possible to obtain more accurate channel state information by combining conventional estimators with CS-based techniques. Moreover, we illustrate that even simple power control techniques can enhance the device detection performance in mMTC setups. Second, we devise a new non-coherent transmission scheme for mMTC and specifically for grant-free random access. We design an algorithm that jointly detects device activity along with embedded information bits. The approach leverages elements from the approximate message passing (AMP) algorithm, and exploits the structured sparsity introduced by the non-coherent transmission scheme. Our analysis reveals that the proposed approach has superior performance compared with application of the original AMP approach.


Compared with QPPNet #REFR that first proposes the plan-structured network, the first and second differences as compared to TLSTM-Cost also apply.

Also, in terms of performance, FasCo demands much fewer training samples than TLSTMCost, which reduces the training cost significantly.

In addition, FasCo does not need to learn the representation of predicates, which means it can be applied to any type of SQL query.

In contrast, TLSTMCost cannot be applied on complex SQL, such as some queries in TPC-H.

Recently there are some techniques that aim to improve the representations of predicates #OTHEREFR .

However, these embedding models are still plagued by complex predicates, such as the "Like" predicate, and they cannot effectively reduce the training cost on string predicates.

Our FasCo inherits the basic tree-based framework but our techniques including featurization and loss function along the training and inference process are surprisingly effective.

Also, QPPNet learns a model for each operator, which can lead to a long training time since the independent model cannot share information.

In contrast, FasCo learns all the operations in one model.

In addition, FasCo adopts the cardinality calibration which significantly improves the accuracy while QPPNet uses the original plan cardinalities.

There are other problem settings around the cost model, such as the end-to-end optimizer.

Less Is More: Towards Lightweight Cost Estimator For Database Systems [Data Management For Data Science

Abstract: We present FasCo, a simple yet effective learning-based estimator for the cost of executing a database query plan. FasCo uses significantly shorter training time and a lower inference cost than the state-of-the-art approaches, while achieving higher estimation accuracy. The effectiveness of FasCo comes from embedding abundant explicit execution-plan-based features and incorporating a novel technique called cardinality calibration. Extensive experimental results show that FasCo achieves orders of magnitude higher efficiency than the state-of-the-art methods: on the JOB-M benchmark dataset, it cuts off training plans by 98%, reducing training time from more than two days to about eight minutes while entailing better accuracy. Furthermore, in dynamic environments, FasCo can maintain satisfactory accuracy even without retraining, narrowing the gap between learning-based estimators and real systems. CCS CONCEPTS • Information systems → Data management systems; • Computing methodologies → Machine learning.

Plan-Structured Deep Neural Network Models For Query Performance Prediction

Abstract: Query performance prediction, the task of predicting the latency of a query, is one of the most challenging problem in database management systems. Existing approaches rely on features and performance models engineered by human experts, but often fail to capture the complex interactions between query operators and input relations, and generally do not adapt naturally to workload characteristics and patterns in query execution plans. In this paper, we argue that deep learning can be applied to the query performance prediction problem, and we introduce a novel neural network architecture for the task: a plan-structured neural network. Our approach eliminates the need for human-crafted feature selection and automatically discovers complex performance models both at the operator and query plan level. Our novel neural network architecture can match the structure of any optimizer-selected query execution plan and predict its latency with high accuracy. We also propose a number of optimizations that reduce training overhead without sacrificing effectiveness. We evaluated our techniques on various workloads and we demonstrate that our plan-structured neural network can outperform the state-of-the-art in query performance prediction.


The IS does not capture intra-class diversity and lacks correlation with the human judgment of image quality and may produce inconsistent results when compared to human evaluations #REFR .

Multiple efforts were spent to improve and build upon the pix2pix model and overcome its limitations. #OTHEREFR The IS in Eq.

(1) is a measure of the quality and diversity of generated images, based on the KL divergence between the predicted and true class distributions of a pre-trained Inception network #OTHEREFR .

where E x∼pgen is the expectation over the images sampled from the generator, KL refers to the Kullback-Leibler divergence, p(y) is the marginal distribution of class labels, and (p(y | x) represents the conditional distribution of class labels y given an input image x.

Although used commonly, The IS has some limitations.

Moreover, it is insensitive to the prior distribution over labels (hence is biased towards ImageNet dataset and Inception model), and is very sensitive to model parameters and implementations #OTHEREFR ).

The IS also requires a large sample size to be reliable.

Similar to IS, the FID depends on the Inception model to generate its value, the difference, however, is that the FID calculates the Wasserstein-2 (a.k.a Fréchet) distance between multivariate Gaussians fitted to the embedding space of the Inception-v3 network of generated and real images #OTHEREFR . The FID is Denoted as:

In this equation, µ g and µ r represent the means of the feature maps of the generated images and real images, respectively, and C g and C r represent the covariance matrices of the feature maps of the generated images and real images, respectively.

Pseudo Supervised Metrics: Evaluating Unsupervised Image To Image Translation Models In Unsupervised Cross-Domain Classification Frameworks

Abstract: Highlights • New metric for evaluating UI2I translation models using pseudo supervised metrics that is designed specifically to support cross-domain classification. • The metric not only outperforms unsupervised metrics such as the FID, but is also highly correlated with the true supervised metrics, robust, and explainable. • We demonstrate that this metric can be used for future research in this field by applying it to a critical real-world problem (the boiling crisis problem).

A Note On The Inception Score

Abstract: Deep generative models are powerful tools that have produced impressive results in recent years. These advances have been for the most part empirically driven, making it essential that we use high quality evaluation metrics. In this paper, we provide new insights into the Inception Score, a recently proposed and widely used evaluation metric for generative models, and demonstrate that it fails to provide useful guidance when comparing models. We discuss both suboptimalities of the metric itself and issues with its application. Finally, we call for researchers to be more systematic and careful when evaluating and comparing generative models, as the advancement of the field depends upon it.


In classical transduction #REFR , given data points {x i } l i=1 and their labels {y i } l i=1 , the objective is making predictions for points {x i } l+k i=l+1 .

I.e., when there exists some X ⊂ X such that P x∼Dtest [x ∈ X ] > 0, but P x∼Dtrain [x ∈ X ] = 0 (see Fig 2b) .

We term the problem of achieving low risk on such a D test as OOS extrapolation.

Out-of-Combination (OOC). This is a special case of OOS.

Let X = X 1 × X 2 be the product of two spaces.

Let D train,X1 , D train,X2 denote the marginal distributions of Transduction.

In this paper we refer to predictors of x that are functions of the labeled data points {x i } l i=1 as transductive predictors.

Abstract: Machine learning systems, especially with overparameterized deep neural networks, can generalize to novel test instances drawn from the same distribution as the training data. However, they fare poorly when evaluated on out-of-support test points. In this work, we tackle the problem of developing machine learning systems that retain the power of overparameterized function approximators while enabling extrapolation to out-of-support test points when possible. This is accomplished by noting that under certain conditions, a "transductive" reparameterization can convert an out-of-support extrapolation problem into a problem of within-support combinatorial generalization. We propose a simple strategy based on bilinear embeddings to enable this type of combinatorial generalization, thereby addressing the out-of-support extrapolation problem under certain conditions. We instantiate a simple, practical algorithm applicable to various supervised learning and imitation learning tasks.

Learning By Transduction

Abstract: We describe a method for predicting a classification of an object given classifications of the objects in the training set, assuming that the pairs object/classification are generated by an i.i.d. process from a continuous probability distribution. Our method is a modification of Vapnik's support-vector machine; its main novelty is that it gives not only the prediction itself but also a practicable measure of the evidence found in support of that prediction. We also describe a procedure for assigning degrees of confidence to predictions made by the support vector machine. Some experimental results are presented, and possible extensions of the algorithms are discussed.


We show below that the similar values of p k lead to the same log 2 n ± O #REFR time it takes to inform a constant fraction of the nodes.

Interestingly, the expansion phase of the pull protocol (when viewed from a distance) resembles the expansion phase of the push protocol-the probability that an uninformed node becomes informed in a round starting with k informed nodes is p k = k n and thus, for small k, very close to the k n − Θ( k 2 n 2 ) probability of the push protocol. Nevertheless, the precise processes are very different.

For example, in the push protocol we almost surely observe a perfect doubling of the number of informed nodes as long as o( √ n) nodes are informed.

For the pull protocol, the number of newly informed nodes in the first round is binomially distributed with parameters n − 1 and 1 n , so the probability for a perfect doubling is asymptotically equal to 1 e .

For this reason, the existing analyses of the push protocol cannot easily be transferred to the pull protocol.

This is different for our method, which ignored many details of the process and only relies on the rough characteristics p k and c k of the process.

From that point on, the double exponential shrinking conditions are obvious, leading to a double logarithmic remaining time.

Randomized Rumor Spreading Revisited

Abstract: We develop a simple and generic method to analyze randomized rumor spreading processes in fully connected networks. In contrast to all previous works, which heavily exploit the precise definition of the process under investigation, we only need to understand the probability and the covariance of the events that uninformed nodes become informed. This universality allows us to easily analyze the classic push, pull, and push-pull protocols both in their pure version and in several variations such as messages failing with constant probability or nodes calling a random number of others each round. Some dynamic models can be analyzed as well, e.g., when the network is a G(n, p) random graph sampled independently each round [Clementi et al. (ESA 2013)]. Despite this generality, our method determines the expected rumor spreading time precisely apart from additive constants, which is more precise than almost all previous works. We also prove tail bounds showing that a deviation from the expectation by more than an additive number of r rounds occurs with probability at most exp(−Ω(r)). We further use our method to discuss the common assumption that nodes can answer any number of incoming calls. We observe that the restriction that only one call can be answered leads to a significant increase of the runtime of the push-pull protocol. In particular, the double logarithmic end phase of the process now takes logarithmic time. This also increases the message complexity from the asymptotically optimal Θ(n log log n) [Karp, Shenker, Schindelhauer, Vöcking (FOCS 2000)] to Θ(n log n). We propose a simple variation of the push-pull protocol that reverts back to the double logarithmic end phase and thus to the Θ(n log log n) message complexity.

Emergence Of Scaling In Random Networks

Abstract: Systems as diverse as genetic networks or the World Wide Web are best described as networks with complex topology. A common property of many large networks is that the vertex connectivities follow a scale-free power-law distribution. This feature was found to be a consequence of two generic mechanisms: (i) networks expand continuously by the addition of new vertices, and (ii) new vertices attach preferentially to sites that are already well connected. A model based on these two ingredients reproduces the observed stationary scale-free distributions, which indicates that the development of large networks is governed by robust self-organizing phenomena that go beyond the particulars of the individual systems.


It is heavily one-sided #REFR , as will be observed with the results in this work.

2 In this work, we define social bias as the unbalanced disposition (or prejudice) in favor of or against a thing, person or group, relative to another, in a way that is deemed as unfair #OTHEREFR .

3 This is harmful bias and it is related to fairness.

In some quarters, bias also involves overgeneralization #OTHEREFR , fulfilling characteristic 2 in the next paragraph.

As a motivation, we address the challenge of estimating bias in text data from some of the many axes (or dimensions) of bias (e.g. race and gender).

Social bias in text usually has some of the following characteristics: 3 1.

It uses extreme or inappropriate language #OTHEREFR .

This forms the basis of the assumption (for some of the samples) in the two datasets used to create the new multi-axes bias dataset (MAB), as discussed in Section 3.

It is based on unsupported or unsubstantiated claims, such as stereotypes #OTHEREFR .

Bipol: A Novel Multi-Axes Bias Evaluation Metric With Explainability For Nlp

Abstract: We introduce bipol, a new metric with explainability, for estimating social bias in text data. Harmful bias is prevalent in many online sources of data that are used for training machine learning (ML) models. In a step to address this challenge we create a novel metric that involves a two-step process: corpuslevel evaluation based on model classification and sentence-level evaluation based on (sensitive) term frequency (TF). After creating new models to detect bias along multiple axes using SotA architectures, we evaluate two popular NLP datasets (COPA and SQuADv2). As additional contribution, we created a large dataset (with almost 2 million labelled samples) for training models in bias detection and make it publicly available. We also make public our codes. 1

Gender Bias In Coreference Resolution: Evaluation And Debiasing Methods

Abstract: We introduce a new benchmark, WinoBias, for coreference resolution focused on gender bias. Our corpus contains Winograd-schema style sentences with entities corresponding to people referred by their occupation (e.g. the nurse, the doctor, the carpenter). We demonstrate that a rule-based, a feature-rich, and a neural coreference system all link gendered pronouns to pro-stereotypical entities with higher accuracy than anti-stereotypical entities, by an average difference of 21.1 in F1 score. Finally, we demonstrate a data-augmentation approach that, in combination with existing word-embedding debiasing techniques, removes the bias demonstrated by these systems in WinoBias without significantly affecting their performance on existing coreference benchmark datasets. Our dataset and code are available at http://winobias.org.


For heterogeneous data fusion between LiDAR and IMU, we introduce the Transformer #REFR encoder architecture.

Although learning-based frameworks offer several advantages such as non-calibration and initialization, robust feature extraction, and reduced time consumption, How should we perform fusion among different modalities in a supervised sensor fusion odometry estimation task? is still one of the key issues remains, which we introduce and comprehensively evaluate in this study.

In this work, we focus on the fusion module in the learningbased multi-modal fusion field, which is also a general component of tasks that uses multiple modalities as input.

In contrast to operate simple element-wise summation and concatenation, the proposed fusion strategy generates the adaptive weights or attentions between different modalities.

The overview of the proposed TransFusionOdom is shown in Fig. 1 .

To achieve homogeneous data fusion between LiDAR's vertex and normal estimation, we implement the Soft Mask Attentional Fusion (SMAF) #OTHEREFR .

This multi-attention fusion approach is designed to avoid the overfitting problem, which is prone to occur if we overly increase the complexity of Transformer-based fusion network #OTHEREFR .

The main contributions of this article can be summarized as follows:

1) An end-to-end Transformer-based supervised multimodal fusion odometry estimation network (TransFu-sionOdom) is developed, which achieves the competitive or even superior performance on KITTI dataset.

The multi-attention fusion module is proposed to achieve the adaptive weights learning for the fusion between homogeneous and heterogeneous modalities.

2) A general visualization approach is demonstrated the interactions between two modalities inside the Transformer architecture, which enhances the interpretability of the proposed learning-based solution.

Transfusionodom: Interpretable Transformer-Based Lidar-Inertial Fusion Odometry Estimation

Abstract: Multi-modal fusion of sensors is a commonly used approach to enhance the performance of odometry estimation, which is also a fundamental module for mobile robots. However, the question of how to perform fusion among different modalities in a supervised sensor fusion odometry estimation task? is still one of challenging issues remains. Some simple operations, such as element-wise summation and concatenation, are not capable of assigning adaptive attentional weights to incorporate different modalities efficiently, which make it difficult to achieve competitive odometry results. Recently, the Transformer architecture has shown potential for multi-modal fusion tasks, particularly in the domains of vision with language. In this work, we propose an end-to-end supervised Transformer-based LiDAR-Inertial fusion framework (namely TransFusionOdom) for odometry estimation. The multi-attention fusion module demonstrates different fusion approaches for homogeneous and heterogeneous modalities to address the overfitting problem that can arise from blindly increasing the complexity of the model. Additionally, to interpret the learning process of the Transformer-based multi-modal interactions, a general visualization approach is introduced to illustrate the interactions between modalities. Moreover, exhaustive ablation studies evaluate different multi-modal fusion strategies to verify the performance of the proposed fusion strategy. A synthetic multi-modal dataset is made public to validate the generalization ability of the proposed fusion strategy, which also works for other combinations of different modalities. The quantitative and qualitative odometry evaluations on the KITTI dataset verify the proposed TransFusionOdom could achieve superior performance compared with other related works.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


Using Mixup Only In The Early Stage Of Training

We note that our results here can be regarded as a complement to #REFR , where the authors show that regularization techniques only matter during the early phase of learning.

Such a conclusion implies that turning of Mixup after a certain number of epochs and returning to standard ERM training may prevent the training from overfitting the noises induced by Mixup.

We now present results obtained from such a training scheme on both CIFAR10 and SVHN in Figure 6 .

Results in Figure 6 clearly indicate that switching from Mixup to ERM at an appropriate time will successfully avoid the generalization degradation.

Figure 6 also suggests that switching Mixup to ERM too early may not boost the model performance.

In addition, if the switch is too late, memorization of noisy data may already taken effect, which impact generalization negatively.

Over-Training With Mixup May Hurt Generalization

Abstract: Mixup, which creates synthetic training instances by linearly interpolating random sample pairs, is a simple and yet effective regularization technique to boost the performance of deep models trained with SGD. In this work, we report a previously unobserved phenomenon in Mixup training: on a number of standard datasets, the performance of Mixup-trained models starts to decay after training for a large number of epochs, giving rise to a U-shaped generalization curve. This behavior is further aggravated when the size of original dataset is reduced. To help understand such a behavior of Mixup, we show theoretically that Mixup training may introduce undesired data-dependent label noises to the synthesized data. Via analyzing a least-square regression problem with a random feature model, we explain why noisy labels may cause the U-shaped curve to occur: Mixup improves generalization through fitting the clean patterns at the early training stage, but as training progresses, Mixup becomes over-fitting to the noise in the synthetic data. Extensive experiments are performed on a variety of benchmark datasets, validating this explanation.

Time Matters In Regularizing Deep Networks: Weight Decay And Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence

Abstract: Regularization is typically understood as improving generalization by altering the landscape of local extrema to which the model eventually converges. Deep neural networks (DNNs), however, challenge this view: We show that removing regularization after an initial transient period has little effect on generalization, even if the final loss landscape is the same as if there had been no regularization. In some cases, generalization even improves after interrupting regularization. Conversely, if regularization is applied only after the initial transient, it has no effect on the final solution, whose generalization gap is as bad as if regularization never happened. This suggests that what matters for training deep networks is not just whether or how, but when to regularize. The phenomena we observe are manifest in different datasets (CIFAR-10, CIFAR-100), different architectures (ResNet-18, All-CNN), different regularization methods (weight decay, data augmentation), different learning rate schedules (exponential, piece-wise constant). They collectively suggest that there is a ``critical period'' for regularizing deep networks that is decisive of the final performance. More analysis should, therefore, focus on the transient rather than asymptotic behavior of learning.


Lemma C.1 s = 3 is the theoretical minimum number of samples for US-Net #REFR .

First, from #OTHEREFR we know sandwich rules: Performances at all widths are bounded by the performance of the model at the smallest and the largest width.

In other words, optimizing the lower and upper bounds of performance can implicitly optimize all sub-networks in a US-Net.

To optimize for arbitrary widths, we need at least one randomly sampled width per iteration, except for the largest and smallest sub-networks.

In conclusion, s = 3 is the theoretical minimum number of samples for US-Net. □

Three Guidelines You Should Know For Universally Slimmable Self-Supervised Learning

Abstract: We propose universally slimmable self-supervised learning (dubbed as US3L) to achieve better accuracy-efficiency trade-offs for deploying self-supervised models across different devices. We observe that direct adaptation of selfsupervised learning (SSL) to universally slimmable networks misbehaves as the training process frequently collapses. We then discover that temporal consistent guidance is the key to the success of SSL for universally slimmable networks, and we propose three guidelines for the loss design to ensure this temporal consistency from a unified gradient perspective. Moreover, we propose dynamic sampling and group regularization strategies to simultaneously improve training efficiency and accuracy. Our US3L method has been empirically validated on both convolutional neural networks and vision transformers. With only once training and one copy of weights, our method outperforms various state-ofthe-art methods (individually trained or not) on benchmarks including recognition, object detection and instance segmentation. Our code is available at https://github.com/ megvii-research/US3L-CVPR2023.

Universally Slimmable Networks And Improved Training Techniques

Abstract: Slimmable networks are a family of neural networks that can instantly adjust the runtime width. The width can be chosen from a predefined widths set to adaptively optimize accuracy-efficiency trade-offs at runtime. In this work, we propose a systematic approach to train universally slimmable networks (US-Nets), extending slimmable networks to execute at arbitrary width, and generalizing to networks both with and without batch normalization layers. We further propose two improved training techniques for US-Nets, named the sandwich rule and inplace distillation, to enhance training process and boost testing accuracy. We show improved performance of universally slimmable MobileNet v1 and MobileNet v2 on ImageNet classification task, compared with individually trained ones and 4-switch slimmable network baselines. We also evaluate the proposed US-Nets and improved training techniques on tasks of image super-resolution and deep reinforcement learning. Extensive ablation experiments on these representative tasks demonstrate the effectiveness of our proposed methods. Our discovery opens up the possibility to directly evaluate FLOPs-Accuracy spectrum of network architectures. Code and models are available at: this https URL


To estimate the unknown parameters in the testing phase, we follow the MCEM optimization scheme by #REFR .

At the Expectation step, the complete-data log-likelihood is approximated by averaging over R samples:

R samples of the latent variable are drawn using the Metropolis-Hastings algorithm with a Gaussian as a symmetric proposal distribution. θ * is an initialization of the parameters.

At the Maximization step, we minimize the loss function, i.e., the negative log-likelihood −RQ(θ; θ * ), with respect to the unknown parameters θ ={HE,WB,HB,R S,f ,R B,f ,gt} using the auxiliary function technique.

For this, equation #OTHEREFR can be viewed as the superposition of a convex function (the first term) and a concave function (the second term), where the former can be bounded using the Jensen's trace inequality and the latter can be bounded using a first-order Taylor expansion #OTHEREFR Appendix A] .

Partially Adaptive Multichannel Joint Reduction Of Ego-Noise And Environmental Noise

Abstract: Human-robot interaction relies on a noise-robust audio processing module capable of estimating target speech from audio recordings impacted by environmental noise, as well as self-induced noise, so-called ego-noise. While external ambient noise sources vary from environment to environment, ego-noise is mainly caused by the internal motors and joints of a robot. Egonoise and environmental noise reduction are often decoupled, i.e., ego-noise reduction is performed without considering environmental noise. Recently, a variational autoencoder (VAE)-based speech model has been combined with a fully adaptive non-negative matrix factorization (NMF) noise model to recover clean speech under different environmental noise disturbances. However, its enhancement performance is limited in adverse acoustic scenarios involving, e.g. ego-noise. In this paper, we propose a multichannel partially adaptive scheme to jointly model ego-noise and environmental noise utilizing the VAE-NMF framework, where we take advantage of spatially and spectrally structured characteristics of ego-noise by pre-training the ego-noise model, while retaining the ability to adapt to unknown environmental noise. Experimental results show that our proposed approach outperforms the methods based on a completely fixed scheme and a fully adaptive scheme when ego-noise and environmental noise are present simultaneously.

Semi-Supervised Multichannel Speech Enhancement With Variational Autoencoders And Non-Negative Matrix Factorization

Abstract: In this paper we address speaker-independent multichannel speech enhancement in unknown noisy environments. Our work is based on a well-established multichannel local Gaussian modeling framework. We propose to use a neural network for modeling the speech spectro-temporal content. The parameters of this supervised model are learned using the framework of variational autoencoders. The noisy recording environment is supposed to be unknown, so the noise spectro-temporal modeling remains unsupervised and is based on non-negative matrix factorization (NMF). We develop a Monte Carlo expectation-maximization algorithm and we experimentally show that the proposed approach outperforms its NMF-based counterpart, where speech is modeled using supervised NMF.


FL, a new generation of technology, is applied in finance, healthcare, and security to address isolated data islands and privacy protection issues #REFR .

In compliance with data security management, each participant in the federated model is able to use its own data for joint training with others in a safe and fair manner without causing data leakage.

In the metaverse, each user can be abstracted as a data owner.

Utilizing all the potential value that may be found in data and fully optimizing business technology and services are both possible with the help of big data and FL technology.

FATE is the first industrial-level federated learning open-source framework developed by WeBank, which can build federated models of general business scenarios.

Federated Learning For Metaverse: A Survey

Abstract: The metaverse, which is at the stage of innovation and exploration, faces the dilemma of data collection and the problem of private data leakage in the process of development. This can seriously hinder the widespread deployment of the metaverse. Fortunately, federated learning (FL) is a solution to the above problems. FL is a distributed machine learning paradigm with privacy-preserving features designed for a large number of edge devices. Federated learning for metaverse (FL4M) will be a powerful tool. Because FL allows edge devices to participate in training tasks locally using their own data, computational power, and model-building capabilities. Applying FL to the metaverse not only protects the data privacy of participants but also reduces the need for high computing power and high memory on servers. Until now, there have been many studies about FL and the metaverse, respectively. In this paper, we review some of the early advances of FL4M, which will be a research direction with unlimited development potential. We first introduce the concepts of metaverse and FL, respectively. Besides, we discuss the convergence of key metaverse technologies and FL in detail, such as big data, communication technology, the Internet of Things, edge computing, blockchain, and extended reality. Finally, we discuss some key challenges and promising directions of FL4M in detail. In summary, we hope that our up-to-date brief survey can help people better understand FL4M and build a fair, open, and secure metaverse. CCS CONCEPTS • Computing methodologies → Federated learning.

A Survey On Federated Learning Systems: Vision, Hype And Reality For Data Privacy And Protection

Abstract: Federated learning has been a hot research topic in enabling the collaborative training of machine learning models among different organizations under the privacy restrictions. As researchers try to support more machine learning models with different privacy-preserving approaches, there is a requirement in developing systems and infrastructures to ease the development of various federated learning algorithms. Similar to deep learning systems such as PyTorch and TensorFlow that boost the development of deep learning, federated learning systems (FLSs) are equivalently important, and face challenges from various aspects such as effectiveness, efficiency, and privacy. In this survey, we conduct a comprehensive review on federated learning systems. To achieve smooth flow and guide future research, we introduce the definition of federated learning systems and analyze the system components. Moreover, we provide a thorough categorization for federated learning systems according to six different aspects, including data distribution, machine learning model, privacy mechanism, communication architecture, scale of federation and motivation of federation. The categorization can help the design of federated learning systems as shown in our case studies. By systematically summarizing the existing federated learning systems, we present the design factors, case studies, and future research opportunities.


For example, by applying a method known as lip reading, images of the mouth of the speaker or the entire face are captured by a camera, and the content of the utterance is estimated from those images #REFR .

However, as for the speech interface, two challenges must be overcome. First, using the interface in public places presents limitations.

In addition to being an annoyance to the surrounding people, disclosing personal information or secret information by uttering it in public is risky in terms of information security.

Next, it cannot be used in a noisy environment, because the accuracy of speech recognition may be declined.

These issues are particularly acute when trying to use a speech interface to interact with wearables or mobile computers.

To overcome these challenges, research on "silent-speech recognition" has been conducted #OTHEREFR .

If the user could simply mouth the utterance without actually voicing it, it would be possible to use such a voice interaction in public places.

However, with the camera method, it is necessary to install a camera in front of the face, and its form factor renders it unsuitable for wearables or mobile applications.

Other approaches of the studies on "non-audible murmur" (NAM) #OTHEREFR attempts to recognize utterances with a microphone or an accelerometer worn on the skin or throat of the user.

In this case, the user speaks with articulated respiratory sound without vocal-fold vibration (namely, whispering).

However, to be recognized accurately by the system, a user's whispering voice tends to be noticeable by other people nearby.

Sottovoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks

Abstract: Computing methodologies → Neural networks; • Humancentered computing → Interface design prototyping; Mobile devices; Sound-based input / output; • Social and professional topics → People with disabilities.

Lipreading With Long Short-Term Memory

Abstract: Lipreading, i.e. speech recognition from visual-only recordings of a speaker's face, can be achieved with a processing pipeline based solely on neural networks, yielding significantly better accuracy than conventional methods. Feed-forward and recurrent neural network layers (namely Long Short-Term Memory; LSTM) are stacked to form a single structure which is trained by back-propagating error gradients through all the layers. The performance of such a stacked network was experimentally evaluated and compared to a standard Support Vector Machine classifier using conventional computer vision features (Eigenlips and Histograms of Oriented Gradients). The evaluation was performed on data from 19 speakers of the publicly available GRID corpus. With 51 different words to classify, we report a best word accuracy on held-out evaluation speakers of 79.6% using the end-to-end neural network-based solution (11.6% improvement over the best feature-based solution evaluated).


We take a step towards explaining this observation via a smoothed-analysis result #REFR Teng, 2004, Sankar et al., 2006] .

While the above results focus on the single vector Krylov method, our bounds naturally generalize to other small block sizes between 1 and k (e.g. 4, 10, or k − 1).

For general small block size b, we show that dependence on g min can be replaced with a dependence on the smallest "b th order" gap g min,b := min i∈{1,...,k−b}

While a dependence on singular value gaps is unavoidable for small block methods in the worst-case, the parameter seems to rarely have an impact in practice.

Specifically, we leverage work in random matrix theory on eigenvalue repulsion, which shows that small spectral gaps in a matrix are brittle: adding a tiny amount of random noise to any matrix ensures that its singular value gaps are at worst inverse polynomial in the problem parameters.

Using this fact, we present bounds that replace the dependence on log(1/g min ) in our prior results with a dependence on log( dκ k δε ) for randomly perturbed matrices, where κ k = σ 1 /σ k measures the conditioning of the top k singular values.

From an algorithm design perspective, the log(1/g min ) can be removed even in the worst case by explicitly adding a random diagonal perturbation to A.

On The Unreasonable Effectiveness Of Single Vector Krylov Methods For Low-Rank Approximation

Abstract: Krylov subspace methods are a ubiquitous tool for computing near-optimal rank k approximations of large matrices. While "large block" Krylov methods with block size at least k give the best known theoretical guarantees, block size one (a single vector) or a small constant is often preferred in practice. Despite their popularity, we lack theoretical bounds on the performance of such "small block" Krylov methods for low-rank approximation. We address this gap between theory and practice by proving that small block Krylov methods essentially match all known low-rank approximation guarantees for large block methods. Via a black-box reduction we show, for example, that the standard single vector Krylov method run for t iterations obtains the same spectral norm and Frobenius norm error bounds as a Krylov method with block size ≥ k run for O(t/) iterations, up to a logarithmic dependence on the smallest gap between sequential singular values. That is, for a given number of matrix-vector products, single vector methods are essentially as effective as any choice of large block size. By combining our result with tail-bounds on eigenvalue gaps in random matrices, we prove that the dependence on the smallest singular value gap can be eliminated if the input matrix is perturbed by a small random matrix. Further, we show that single vector methods match the more complex algorithm of [Bakshi et al. '22], which combines the results of multiple block sizes to achieve an improved algorithm for Schatten p-norm low-rank approximation.

Smoothed Analysis Of The Condition Numbers And Growth Factors Of Matrices

Abstract: Let $\orig{A}$ be any matrix and let $A$ be a slight random perturbation of $\orig{A}$. We prove that it is unlikely that $A$ has large condition number. Using this result, we prove it is unlikely that $A$ has large growth factor under Gaussian elimination without pivoting. By combining these results, we bound the smoothed precision needed by Gaussian elimination without pivoting. Our results improve the average-case analysis of Gaussian elimination without pivoting performed by Yeung and Chan (SIAM J. Matrix Anal. Appl., 1997).


We can refine our semantics of Logical SDQLite by interpreting our types as generalizations of Cartesian spaces, called diffeological spaces [Iglesias-Zemmour 2013] , and our functions as morphisms of diffeological spaces, which can be thought of as generalized smooth functions between these spaces, closely following #REFR .

The correctness of the rewrite rules from Figure 9 is immediate, as they can all be seen as special cases of optimizations given in related work #OTHEREFR

2022; , and they rely on the semi-ring structure of our dictionaries.

The limitation of the non-smooth semantics is that it does not inform us of the differentiability of our programs.

A function : R → R is smooth if it is differentiable and its derivative is smooth.

More precisely, a diffeological space is a pair ( , P ) of a set and sets of functions P ⊆ for all subsets of Euclidean spaces R , called plots.

Each such set of plots P must contain all constant functions, be closed under pre-composition by smooth functions, and be closed under countable glueing.

The latter means that if ( : ⊆ R → ) ∈N are a countable family of plots such that for each , ∈ N, and agree on ∩ , then : → , ∈ ↦ → ( ) is a plot.

Morphisms : ( , P ) → ( , P ) between diffeological spaces are functions : → such that for all ∈ P , • ∈ P .

N or B with constant functions as plots form diffeological spaces.

∇Sd: Differentiable Programming For Sparse Tensors

Abstract: Sparse tensors are prevalent in many data-intensive applications, yet existing differentiable programming frameworks are tailored towards dense tensors. This presents a significant challenge for efficiently computing gradients through sparse tensor operations, as their irregular sparsity patterns can result in substantial memory and computational overheads. In this work, we introduce a novel framework that enables the efficient and automatic differentiation of sparse tensors, addressing this fundamental issue. Our experiments demonstrate the effectiveness of the proposed framework in terms of performance and scalability, outperforming stateof-the-art frameworks across a range of synthetic and real-world datasets. Our approach offers a promising direction for enabling efficient and scalable differentiable programming with sparse tensors, which has significant implications for numerous applications in machine learning, natural language processing, and scientific computing.

Correctness Of Automatic Differentiation Via Diffeologies And Categorical Gluing

Abstract: We present semantic correctness proofs of Automatic Differentiation (AD). We consider a forward-mode AD method on a higher order language with algebraic data types, and we characterise it as the unique structure preserving macro given a choice of derivatives for basic operations. We describe a rich semantics for differentiable programming, based on diffeological spaces. We show that it interprets our language, and we phrase what it means for the AD method to be correct with respect to this semantics. We show that our characterisation of AD gives rise to an elegant semantic proof of its correctness based on a gluing construction on diffeological spaces. We explain how this is, in essence, a logical relations argument. Finally, we sketch how the analysis extends to other AD methods by considering a continuation-based method.

generalized smooth functions
diffeological spaces


The method is related to Stochastic Weight Averaging (SWA) #REFR , but we do not use a modified learning rate schedule here because instead of getting better generalization ability, WE aim to give an improved learning-forgetting tradeoff. As shown in Fig.

3) , the model performs better on new tasks while losing accuracy on previous ones.

The models among training compose a sequence of different learning-forgetting tradeoffs.

Instead of interpolating only between the initial and the final model, our method weight ensemble (WE) averages the weights in the sequence during the training time:

where model weight sampling happens every I iteration.

3 , WE achieves better performance on downstream tasks than WiSE-FT.

In addition, while WiSE-FT is sensitive to different values of α, our method is much more robust under different hyperparameter (I) choices.

Preventing Zero-Shot Transfer Degradation In Continual Learning Of Vision-Language Models

Abstract: Continual learning (CL) can help pre-trained visionlanguage models efficiently adapt to new or under-trained data distributions without retraining. Nevertheless, during the continual training of the Contrastive Language-Image Pre-training (CLIP) model, we observe that the model's zero-shot transfer ability significantly degrades due to catastrophic forgetting. Existing CL methods can mitigate forgetting by replaying previous data. However, since the CLIP dataset is private, replay methods cannot access the pre-training dataset. In addition, replaying data of previously learned downstream tasks can enhance their performance but comes at the cost of sacrificing zero-shot performance. To address this challenge, we propose a novel method ZSCL to prevent zero-shot transfer degradation in the continual learning of vision-language models in both feature and parameter space. In the feature space, a reference dataset is introduced for distillation between the current and initial models. The reference dataset should have semantic diversity but no need to be labeled, seen in pre-training, or matched image-text pairs. In parameter space, we prevent a large parameter shift by averaging weights during the training. We propose a more challenging Multi-domain Task Incremental Learning (MTIL) benchmark to evaluate different methods, where tasks are from various domains instead of class-separated in a single dataset. Our method outperforms other methods in the traditional class-incremental learning setting and the MTIL by 9.7% average score. Our code locates at https: //github.com/Thunderbeee/ZSCL.

Averaging Weights Leads To Wider Optima And Better Generalization

Abstract: Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) procedure finds much broader optima than SGD, and approximates the recent Fast Geometric Ensembling (FGE) approach with a single model. Using SWA we achieve notable improvement in test accuracy over conventional SGD training on a range of state-of-the-art residual networks, PyramidNets, DenseNets, and Shake-Shake networks on CIFAR-10, CIFAR-100, and ImageNet. In short, SWA is extremely easy to implement, improves generalization, and has almost no computational overhead.

improved learning-forgetting tradeoff
Stochastic Weight Averaging


In many scenarios, an approximation of the scene geometry is either given, e.g., when using SfM to compute the input camera poses, or can be computed, e.g., via MVS or single-view depth predictions #REFR .

A common strategy in the literature is to represent the scene explicitly through a voxel volume.

In contrast to this uniform subdivision, we investigate using an adaptive subdivision of the scene.

This allows us to compute an adaptive subdivision of the scene via Delaunay triangulation #OTHEREFR of such a point cloud.

This results in a set of non-overlapping tetrahedra, where smaller tetrahedra are created close to the surface of the scene (c.f . Fig. 2 ).

In the following, we explain how this adaptive subdivision of the scene can be used instead of voxels for volume rendering and neural rendering.

Tetra-Nerf: Representing Neural Radiance Fields Using Tetrahedra

Abstract: 2) scene representation: dense tetrahedra 3) barycetric interpolation of features 4) shallow MLP predicts density and colour Figure 1. The input to Tetra-NeRF is a point cloud which is triangulated to get a set of tetrahedra used to represent the radiance field. Rays are sampled, and the field is queried. The barycentric interpolation is used to interpolate tetrahedra vertices, and the resulting features are passed to a shallow MLP to get the density and colours for volumetric rendering.

Monocular Depth Estimation Based On Deep Learning: An Overview

Abstract: Depth information is important for autonomous systems to perceive environments and estimate their own state. Traditional depth estimation methods, like structure from motion and stereo vision matching, are built on feature correspondences of multiple viewpoints. Meanwhile, the predicted depth maps are sparse. Inferring depth information from a single image (monocular depth estimation) is an ill-posed problem. With the rapid development of deep neural networks, monocular depth estimation based on deep learning has been widely studied recently and achieved promising performance in accuracy. Meanwhile, dense depth maps are estimated from single images by deep neural networks in an end-to-end manner. In order to improve the accuracy of depth estimation, different kinds of network frameworks, loss functions and training strategies are proposed subsequently. Therefore, we survey the current monocular depth estimation methods based on deep learning in this review. Initially, we conclude several widely used datasets and evaluation indicators in deep learning-based depth estimation. Furthermore, we review some representative existing methods according to different training manners: supervised, unsupervised and semi-supervised. Finally, we discuss the challenges and provide some ideas for future researches in monocular depth estimation.


C Dataset Details C.1 RECIPEPLAN Data Repurpose RECIPEQA was proposed in #REFR that provide four tasks (Textual Cloze, Visual Cloze, Visual Ordering, Visual Coherence) for multimodal machine comprehension of cooking recipes. This dataset contains question-answer pairs generated from copyrightfree recipes.

We showcase all the details of the outputs of each module for the example task "How to make a candy bouquet" in Figure 8 .

In addition, we showcase examples on "How to make peppermint patty pudding shot" and "How to make bartending for beginners simple margarita" in Figure 9 .

Each of them is under a license, which is provided in each data JSON file.

We collect RECIPEPLAN by repurposing the test dataset from RECIPEQA that relates to the Visual Ordering task as the sequence generation task for the multimodal procedural planning evaluation testbed.

We use recipe instructions as textual plan reference and their paired images as visual plan reference.

Dataset Statistics We visualize two examples of our repurposed RECIPEPLAN for multimodal procedural planning in Figure 10 .

We also show the word-cloud distribution of task name and textual plan reference in Figure 12 .

Multimodal Procedural Planning Via Dual Text-Image Prompting

Abstract: How to make traditional szechuan chicken? Step 5: Add other ingredients like sichuan peppercorn powder and chili bean paste, then stir fry. Step 6: Pour in remaining marinade mixture into the same wok and stir fry until combined. Step 7: Return chicken pieces into the wok and stir fry until all liquid forms a glossy glaze. Step 8: Transfer Szechuan Chicken onto serving plate garnished with green & serve immediately! Step 1: Gather the ingredients such as chicken, garlic, ginger, and light soy sauce. Step 2: Marinade the chicken with a teaspoon of light soy sauce and cooking wine. Step 3: Heat up a wok over medium-high heat until it starts to smoke and add oil. Step 4: Add in marinated chicken pieces and stir fry until they turn golden brown. Figure 1: Our dual Text-Image Prompting (TIP) model generates coherent and authentic multimodal procedural plans with multiple steps towards a high-level goal, providing useful guidelines in task completion.

Recipeqa: A Challenge Dataset For Multimodal Comprehension Of Cooking Recipes

Abstract: Understanding and reasoning about cooking recipes is a fruitful research direction towards enabling machines to interpret procedural text. In this work, we introduce RecipeQA, a dataset for multimodal comprehension of cooking recipes. It comprises of approximately 20K instructional recipes with multiple modalities such as titles, descriptions and aligned set of images. With over 36K automatically generated question-answer pairs, we design a set of comprehension and reasoning tasks that require joint understanding of images and text, capturing the temporal flow of events and making sense of procedural knowledge. Our preliminary results indicate that RecipeQA will serve as a challenging test bed and an ideal benchmark for evaluating machine comprehension systems. The data and leaderboard are available at this http URL


Registration-And Uncertainty-Based Framework For Subject-Level One-Shot White Matter Tract Segmentation

Pseudo subjects and corresponding pseudo labels are generated by deep-learning-based registration using 3D U-Net #REFR .

Our RPA module based on more advanced deep-learning-based registration methods #OTHEREFR can potentially obtain the high-quality pseudo dataset.

We modified registration methods in #OTHEREFR , which are originally for MRI data, to be used for our peak subjects.

Let {x, l} be the only labeled subject and its corresponding tract segmentation label, and y (i) be a set of unlabeled subjects on a spatial domain R 3 .

Pseudo peak subjects and their corresponding labels are synthesized with the RPA module.

Specifically, a spatial transform model is learned to register the labeled subject to unlabeled subjects.

Set g(x, y (i) ) = u (i) is the spatial transform model, where θ are the model parameters, and the output of the model u is the voxel-wise displacement field.

The deformation function ϕ (i) = id + u (i) , where id is the identity transform #OTHEREFR .

Therefore, the pseudo subject x (i) and the corresponding pseudo label l (i) are as followed:

A Registration-And Uncertainty-Based Framework For White Matter Tract Segmentation With Only One Annotated Subject

Abstract: White matter (WM) tract segmentation based on diffusion magnetic resonance imaging (dMRI) plays an important role in the analysis of human health and brain diseases. However, the annotation of WM tracts is time-consuming and needs experienced neuroanatomists. In this study, to explore tract segmentation in the challenging setting of minimal annotations, we propose a novel framework utilizing only one annotated subject (subject-level one-shot) for tract segmentation. Our method is constructed by proposed registration-based peak augmentation (RPA) and uncertaintybased refining (URe) modules. RPA module synthesizes pseudo subjects and their corresponding labels to improve the tract segmentation performance. The proposed URe module alleviates the negative influence of the low-confidence voxels on pseudo subjects. Experimental results show that our method outperforms other state-of-the-art methods by a large margin, and our proposed modules are effective. Overall, our method achieves accurate whole-brain tract segmentation with only one annotated subject. Our code is available at https://github.com/HaoXu0507/ ISBI2023-One-Shot-WM-Tract-Segmentation.

U-Net: Convolutional Networks For Biomedical Image Segmentation

Abstract: There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .


Hence, for this class of models, the word 'interpretability' refers to the ability to clarify and extract knowledge representations in different layers of NN-based models as defined in #REFR .

In the past several years, classical NN-based models have achieved great success and outperform humans in numerous difficult tasks, such as image and visual classification, natural language processing and recognition, and board games.

However, the black-box nature of the classical models presents a real challenge to explain the underlying mechanisms and behaviors of the networks.

The popular solution to address this problem is to explore interpretability via explaining the inside of the black-box.

In this section, the most studied methods in this class, FFNN based and DL based, are surveyed.

Interpretability Of Machine Learning: Recent Advances And Future Prospects

Abstract: The proliferation of machine learning (ML) has drawn unprecedented interest in the study of various multimedia contents such as text, image, audio and video, among others. Consequently, understanding and learning ML-based representations have taken center stage in knowledge discovery in intelligent multimedia research and applications. Nevertheless, the black-box nature of contemporary ML, especially in deep neural networks (DNNs), has posed a primary challenge for ML-based representation learning. To address this black-box problem, the studies on interpretability of ML have attracted tremendous interests in recent years. This paper presents a survey on recent advances and future prospects on interpretability of ML, with several application examples pertinent to multimedia computing, including text-image cross-modal representation learning, face recognition, and the recognition of objects. It is evidently shown that the study of interpretability of ML promises an important research direction, one which is worth further investment in.

Interpretable Convolutional Neural Networks

Abstract: This paper proposes a method to modify a traditional convolutional neural network (CNN) into an interpretable CNN, in order to clarify knowledge representations in high conv-layers of the CNN. In an interpretable CNN, each filter in a high conv-layer represents a specific object part. Our interpretable CNNs use the same training data as ordinary CNNs without a need for any annotations of object parts or textures for supervision. The interpretable CNN automatically assigns each filter in a high conv-layer with an object part during the learning process. We can apply our method to different types of CNNs with various structures. The explicit knowledge representation in an interpretable CNN can help people understand the logic inside a CNN, i.e. what patterns are memorized by the CNN for prediction. Experiments have shown that filters in an interpretable CNN are more semantically meaningful than those in a traditional CNN. The code is available at https://github.com/zqs1022/interpretableCNN.


The coefficients a 0 , a 1 , and a 2 are derived by satisfying boundary conditions #REFR . Fig. 2 shows visual examples of a sample augmentation.

We leverage geometric data augmentation #OTHEREFR to improve model generalization performance by learning geometric invariance.

Each sample is augmented by random rotation and translation, and a polynomial warping function applied on the dense maps and observed trajectories

where ξ is a substitute for spatial coordinates i and j, and ξ denotes warped coordinates.

We create dense warp maps by using the inverse function of (1) to map each warped coordinate ξ to an original coordinate ξ.

Learning To Predict Navigational Patterns From Partial Observations

Abstract: Human beings cooperatively navigate ruleconstrained environments by adhering to mutually known navigational patterns, which may be represented as directional pathways or road lanes. Inferring these navigational patterns from incompletely observed environments is required for intelligent mobile robots operating in unmapped locations. However, algorithmically defining these navigational patterns is nontrivial. This paper presents the first self-supervised learning (SSL) method for learning to infer navigational patterns in realworld environments from partial observations only. We explain how geometric data augmentation, predictive world modeling, and an information-theoretic regularizer enables our model to predict an unbiased local directional soft lane probability (DSLP) field in the limit of infinite data. We demonstrate how to infer global navigational patterns by fitting a maximum likelihood graph to the DSLP field. Experiments show that our SSL model outperforms two SOTA supervised lane graph prediction models on the nuScenes dataset. We propose our SSL method as a scalable and interpretable continual learning paradigm for navigation by perception. Code released upon publication.

Learning A Directional Soft Lane Affordance Model For Road Scenes Using Self-Supervision

Abstract: Humans navigate complex environments in an organized yet flexible manner, adapting to the context and implicit social rules. Understanding these naturally learned patterns of behavior is essential for applications such as autonomous vehicles. However, algorithmically defining these implicit rules of human behavior remains difficult. This work proposes a novel self-supervised method for training a probabilistic network model to estimate the regions humans are most likely to drive in as well as a multimodal representation of the inferred direction of travel at each point. The model is trained on individual human trajectories conditioned on a representation of the driving environment. The model is shown to successfully generalize to new road scenes, demonstrating potential for real-world application as a prior for socially acceptable driving behavior in challenging or ambiguous scenarios which are poorly handled by explicit traffic rules.


Results Of Training The Distortion Estimator

By contrast, our distortion estimator has two regressors for the focal length f and distortion coefficient k1. The input image is generated from #REFR .

We report the performance of our distortion estimator to describe the difference between Wakai's method #OTHEREFR and the distortion estimator.

Our distortion estimator is composed of Wakai's calibration network #OTHEREFR without the tilt Figure 10 : Network architectures of (a) Wakai's calibration network #OTHEREFR and (b) our distortion network.

Wakai's calibration network predicts extrinsics (tilt θ and roll ψ angles) and intrinsics (focal length f and a distortion coefficient k1).

and roll angle regressors, as shown in Figure 10 .

We optimized the distortion estimator after pretraining on Wakai's calibration network.

The distortion estimator achieved slight improvements in the focal length f and distortion coefficient k 1 because the number of predicted camera parameters was reduced by two, that is, the tilt and roll angles, as shown in Table 12 .

Deep Single Image Camera Calibration By Heatmap Regression To Recover Fisheye Images Under Manhattan World Assumption Without Ambiguity

Abstract: In orthogonal world coordinates, a Manhattan world lying along cuboid buildings is widely useful for various computer vision tasks. However, the Manhattan world has much room for improvement because the origin of pan angles from an image is arbitrary, that is, four-fold rotational symmetric ambiguity of pan angles. To address this problem, we propose a definition for the pan-angle origin based on the directions of the roads with respect to a camera and the direction of travel. We propose a learning-based calibration method that uses heatmap regression to remove the ambiguity by each direction of labeled image coordinates, similar to pose estimation keypoints. Simultaneously, our two-branched network recovers the rotation and removes fisheye distortion from a general scene image. To alleviate the lack of vanishing points in images, we introduce auxiliary diagonal points that have the optimal 3D arrangement of spatial uniformity. Extensive experiments demonstrated that our method outperforms conventional methods on large-scale datasets and with off-the-shelf cameras.

The Streetlearn Environment And Dataset

Abstract: Navigation is a rich and well-grounded problem domain that drives progress in many different areas of research: perception, planning, memory, exploration, and optimisation in particular. Historically these challenges have been separately considered and solutions built that rely on stationary datasets - for example, recorded trajectories through an environment. These datasets cannot be used for decision-making and reinforcement learning, however, and in general the perspective of navigation as an interactive learning task, where the actions and behaviours of a learning agent are learned simultaneously with the perception and planning, is relatively unsupported. Thus, existing navigation benchmarks generally rely on static datasets (Geiger et al., 2013; Kendall et al., 2015) or simulators (Beattie et al., 2016; Shah et al., 2018). To support and validate research in end-to-end navigation, we present StreetLearn: an interactive, first-person, partially-observed visual environment that uses Google Street View for its photographic content and broad coverage, and give performance baselines for a challenging goal-driven navigation task. The environment code, baseline agent code, and the dataset are available at this http URL


For MedNLI, we compare with the best transfer learning results so far, ESIM (MultiNLI) which was trained on MultiNLI datasets #REFR .

Models only evaluated on NLI do not require summarisation data, hence "−".

"∼" indicates that BERT and CXR-BERT were fine-tuned on MedNLI, a 'neardomain' NLI dataset.

BERT -Miura et al. ∼ / − CXR-BERT -Boecking et al. ∼ / − ESIM -Chen et al.

/ − T0 & T0++ / GPT-3 / GPT-3-{NLI, GW} / CXR-BERT-NLI / − PubMedBERT-NLI / − SciFive large -{NLI, GW} / T5 large -MLM→Task / DOT5 /

For radiology summarisation, to our knowledge, we are the first to report results on direct transfer from general-domain summarisation.

For biomedical summarisation, since we use a new dataset (PubMed ShortSum), there is no prior comparison.

Large language models T0 #OTHEREFR and GPT-3 (Brown et al., 2020) are massively pretrained language models that can be used off-theshelf for zero-shot or few-shot inference.

T0 is pretrained with multiple tasks including generaldomain summarisation datasets (but not NLI), and shows strong transfer ability #OTHEREFR .

T0 can be seen as a strong general-domain summarisation model and also strong zero-shot domain transfer baseline on summarisation.

Compositional Zero-Shot Domain Transfer With Text-To-Text Models

Abstract: Label scarcity is a bottleneck for improving task performance in specialised domains. We propose a novel compositional transfer learning framework (DOT5 1) for zeroshot domain transfer. Without access to in-domain labels, DOT5 jointly learns domain knowledge (from masked language modelling of unlabelled in-domain free text) and task knowledge (from task training on more readily available general-domain data) in a multi-task manner. To improve the transferability of task training, we design a strategy named NLGU: we simultaneously train natural language generation (NLG) for indomain label-to-data generation which enables data augmentation for self-finetuning and natural language understanding (NLU) for label prediction. We evaluate DOT5 on the biomedical domain and the resource-lean subdomain of radiology, focusing on natural language inference, text summarisation and embedding learning. DOT5 demonstrates the effectiveness of compositional transfer learning through multi-task learning. In particular, DOT5 outperforms the current stateof-the-art in zero-shot transfer by over 7 absolute points in accuracy on RadNLI. We validate DOT5 with ablations and a case study demonstrating its ability to solve challenging NLI examples requiring in-domain expertise. * Work done at Microsoft Health Futures. 1 DOT5 (read as "dot five"): Domain Compositional ZerOshot T5.

Lessons From Natural Language Inference In The Clinical Domain

Abstract: State of the art models using deep neural networks have become very good in learning an accurate mapping from inputs to outputs. However, they still lack generalization capabilities in conditions that differ from the ones encountered during training. This is even more challenging in specialized, and knowledge intensive domains, where training data is limited. To address this gap, we introduce MedNLI - a dataset annotated by doctors, performing a natural language inference task (NLI), grounded in the medical history of patients. We present strategies to: 1) leverage transfer learning using datasets from the open domain, (e.g. SNLI) and 2) incorporate domain knowledge from external data and lexical sources (e.g. medical terminologies). Our results demonstrate performance gains using both strategies.