Similarly, deep convolutional neural networks have been shown to achieve superior performance in detecting pneumonia and other pathologies from chest X-rays, compared to practicing radiologists #REFR .

Academic hospitals are increasingly dedicating resources to bring machine learning (ML) to the bedside and to addressing issues encountered by clinical staff.

These resources are being utilized across a range of applications including clinical decision support, early warning, treatment recommendation, risk prediction, image informatics, telediagnosis, drug discovery, and intelligent health knowledge systems.

There are various examples of ML being applied to medical data, including prediction of sepsis #OTHEREFR , in-hospital mortality, prolonged length-of-stay, patient deterioration, and unplanned readmission #OTHEREFR .

In particular, sepsis is one of the leading causes of in-hospital deaths.

A large-scale study demonstrated the impact of an early warning system to reduce the lead time for detecting the onset of sepsis, and hence allowing more time for clinicians to prescribe antibiotics #OTHEREFR .

These results highlight the potential of ML models when they are strongly integrated into clinical workflows.

When deployed successfully, data-driven models can free time for clinicians #OTHEREFR , improve clinical outcomes #OTHEREFR , reduce costs #OTHEREFR , and provide improved quality care for patients.

However, most studies remain preliminary, limited to small datasets, and/or implemented in select health sub-systems.

Integrating with clinical workflows remains crucial #OTHEREFR but, despite recent computational advances and an explosion of health data, deploying ML in healthcare responsibly and reliably faces several operational and engineering challenges, including:

Mlhops: Machine Learning For Healthcare Operations

Abstract: Machine Learning Health Operations (MLHOps) is the combination of processes for reliable, efficient, usable, and ethical deployment and maintenance of machine learning models in healthcare settings. This paper provides both a survey of work in this area and guidelines for developers and clinicians to deploy and maintain their own models in clinical practice. We cover the foundational concepts of general machine learning operations, describe the initial setup of MLHOps pipelines (including data sources, preparation, engineering, and tools). We then describe long-term monitoring and updating (including data distribution shifts and model updating) and ethical considerations (including bias, fairness, interpretability, and privacy). This work therefore provides guidance across the full pipeline of MLHOps from conception to initial and ongoing deployment.

Chexnet: Radiologist-Level Pneumonia Detection On Chest X-Rays With Deep Learning

Abstract: We develop an algorithm that can detect pneumonia from chest X-rays at a level exceeding practicing radiologists. Our algorithm, CheXNet, is a 121-layer convolutional neural network trained on ChestX-ray14, currently the largest publicly available chest X-ray dataset, containing over 100,000 frontal-view X-ray images with 14 diseases. Four practicing academic radiologists annotate a test set, on which we compare the performance of CheXNet to that of radiologists. We find that CheXNet exceeds average radiologist performance on the F1 metric. We extend CheXNet to detect all 14 diseases in ChestX-ray14 and achieve state of the art results on all 14 diseases.


Both (a) and (b) essentially imply a sort of intrinsic robustness of the ODE flow φ t (z 0 ) to perturbations on z 0 #REFR .

− − → E(T ) generated by H ζ ζ ζ for some time t ∈ [0, T ] and propagator E.

The ansatz is a good choice because of the following two properties of ODE paths: (a) they do not intersect and (b) if paths φ

(A) 0 , φ (B) 0 start close compared to path φ (C) 0 , then paths φ (A) t , φ (B) t remain close compared to path φ (C) t .

Both properties are well known #OTHEREFR for ODEs and become very useful when we try to predict the trajectories from noisy quantum data by imposing strong priors on the space of learnable Hamiltonians.

Property (b) is a consequence of Gronwall's inequality #OTHEREFR and essentially can be interpreted as: ODE flows that start off closer (w.r.t. the initial condition) stay closer (w.r.t. the final condition).

They constrain the trajectories predicted by the model M ζ ζ ζ to be intrinsically robust to small noise in the states s k and inaccuracies in the learned system Hamiltonian H

We call the SAC equipped with this differentiable ODE model the learnable Hamiltonian model-based SAC (LH-MBSAC) as listed in Algorithm 2.

Crucially, we note that LH-MBSAC generalizes the SAC by allowing the policy to interact with the ODE model and the physical system.

LH-MBSAC gracefully falls back to the model-free SAC in the absence of a model with low prediction error that is measured from the performance of the model's predictions on an unseen validation set of interaction data.

Sample-Efficient Model-Based Reinforcement Learning For Quantum Control

Abstract: We propose a model-based reinforcement learning (RL) approach for noisy time-dependent gate optimization with improved sample complexity over model-free RL. Sample complexity is the number of controller interactions with the physical system. Leveraging an inductive bias, inspired by recent advances in neural ordinary differential equations (ODEs), we use an auto-differentiable ODE parametrised by a learnable Hamiltonian ansatz to represent the model approximating the environment whose time-dependent part, including the control, is fully known. Control alongside Hamiltonian learning of continuous time-independent parameters is addressed through interactions with the system. We demonstrate an order of magnitude advantage in the sample complexity of our method over standard model-free RL in preparing some standard unitary gates with closed and open system dynamics, in realistic numerical experiments incorporating single shot measurements, arbitrary Hilbert space truncations and uncertainty in Hamiltonian parameters. Also, the learned Hamiltonian can be leveraged by existing control methods like GRAPE for further gradient-based optimization with the controllers found by RL as initializations. Our algorithm that we apply on nitrogen vacancy (NV) centers and transmons in this paper is well suited for controlling partially characterised one and two qubit systems.

On Robustness Of Neural Ordinary Differential Equations

Abstract: Neural ordinary differential equations (ODEs) have been attracting increasing attention in various research domains recently. There have been some works studying optimization issues and approximation capabilities of neural ODEs, but their robustness is still yet unclear. In this work, we fill this important gap by exploring robustness properties of neural ODEs both empirically and theoretically. We first present an empirical study on the robustness of the neural ODE-based networks (ODENets) by exposing them to inputs with various types of perturbations and subsequently investigating the changes of the corresponding outputs. In contrast to conventional convolutional neural networks (CNNs), we find that the ODENets are more robust against both random Gaussian perturbations and adversarial attack examples. We then provide an insightful understanding of this phenomenon by exploiting a certain desirable property of the flow of a continuous-time ODE, namely that integral curves are non-intersecting. Our work suggests that, due to their intrinsic robustness, it is promising to use neural ODEs as a basic block for building robust deep network models. To further enhance the robustness of vanilla neural ODEs, we propose the time-invariant steady neural ODE (TisODE), which regularizes the flow on perturbed data via the time-invariant property and the imposition of a steady-state constraint. We show that the TisODE method outperforms vanilla neural ODEs and also can work in conjunction with other state-of-the-art architectural methods to build more robust deep networks.


Differentiation Of Forged And Authentic Paa Images

The fool rate of the iterative refinement-based PAA image generation is 49.375%, which is much higher than the fool rate of ESRGAN, which is only 20% #REFR .

We employ a random cropping and shuffling method to create a dataset consisting of 40 PAA images, both authetic and forged.

We then sent this dataset to 14 experts and asked them to differentiate between real and forged images. As shown in Fig.

3(c) , the labels Y and N indicate whether an image is classified as a real and forged PAA image, respectively.

Each red point or black cross represents a result from an expert, and the y-axis refers to the number of samples classified as either Y or N.

Our results showed that the probability of authentic PAA images of human lips being classified as real is 48.04%, while the probability of placing the forged images into the real pool is 46.79%.

To quantitatively compare the forged PAA images and other images with high-resolution PAA images as the standard, we utilized the Fréchet Inception Distance #OTHEREFR , which measures the distance between two images based on the mean and covariance matrices from the Inception V3 model.

Despite not being trained on authentic PAA images, the Inception V3 model can still differentiate between blood vessel image sets using the Fréchet Inception Distance.

The Fréchet Inception Distance between forged and authentic PAA images is

where , stand for the input image and the ground-truth, , are mean matrices of the input and the ground-truth, () is the trace of a matrix, and Σ , Σ are covariance matrices of the input and ground-truth, respectively.

Self-Similarity-Based Super-Resolution Of Photoacoustic Angiography From Hand-Drawn Doodles

Abstract: Deep-learning-based super-resolution photoacoustic angiography (PAA) is a powerful tool that restores blood vessel images from under-sampled images to facilitate disease diagnosis. Nonetheless, due to the scarcity of training samples, PAA super-resolution models often exhibit inadequate generalization capabilities, particularly in the context of continuous monitoring tasks. To address this challenge, we propose a novel approach that employs a super-resolution PAA method trained with forged PAA images. We start by generating realistic PAA images of human lips from hand-drawn curves using a diffusion-based image generation model. Subsequently, we train a self-similarity-based super-resolution model with these forged PAA images. Experimental results show that our method outperforms the super-resolution model trained with authentic PAA images in both original-domain and cross-domain tests. Specially, our approach boosts the quality of super-resolution reconstruction using the images forged by the deep learning model, indicating that the collaboration between deep learning models can facilitate generalization, despite limited initial dataset. This approach shows promising potential for exploring zero-shot learning neural networks for vision tasks.

Model-Based Learning For Accelerated, Limited-View 3-D Photoacoustic Tomography

Abstract: Recent advances in deep learning for tomographic reconstructions have shown great potential to create accurate and high quality images with a considerable speed up. In this paper, we present a deep neural network that is specifically designed to provide high resolution 3-D images from restricted photoacoustic measurements. The network is designed to represent an iterative scheme and incorporates gradient information of the data fit to compensate for limited view artifacts. Due to the high complexity of the photoacoustic forward operator, we separate training and computation of the gradient information. A suitable prior for the desired image structures is learned as part of the training. The resulting network is trained and tested on a set of segmented vessels from lung computed tomography scans and then applied to in-vivo photoacoustic measurement data.


The widely used Hyperledger Fabric #REFR was not considered, as it is based on PoA-based consensus.

We do not depend on smart contracts, which made our selection of a concrete blockchain more flexible. Potential choices included Bitcoin #OTHEREFR and Ethereum #OTHEREFR .

Between the two, Ethereum offers multiple advantages for our use case, namely a configurable hash difficulty and support for the creation of private chains [31] . Therefore, we use it for our implementation.

As the client, we use the official implementation Go Ethereum (Geth).

Note that, at the time of implementation, Ethereum still used PoW consensus. Recently, Ethereum switched to PoS consensus #OTHEREFR .

When setting up a private network, it would therefore have to be configured to use PoW instead. Alternatively, a different PoW-based blockchain can be used.

Decentralized Inverse Transparency With Blockchain

Abstract: Employee data can be used to facilitate work, but their misusage may pose risks for individuals. Inverse transparency therefore aims to track all usages of personal data, allowing individuals to monitor them to ensure accountability for potential misusage. This necessitates a trusted log to establish an agreed-upon and non-repudiable timeline of events. The unique properties of blockchain facilitate this by providing immutability and availability. For power asymmetric environments such as the workplace, permissionless blockchain is especially beneficial as no trusted third party is required. Yet, two issues remain: (1) In a decentralized environment, no arbiter can facilitate and attest to data exchanges. Simple peer-to-peer sharing of data, conversely, lacks the required non-repudiation. (2) With data governed by privacy legislation such as the GDPR, the core advantage of immutability becomes a liability. After a rightful request, an individual's personal data need to be rectified or deleted, which is impossible in an immutable blockchain. To solve these issues, we present Kovacs, a decentralized data exchange and usage logging system for inverse transparency built on blockchain. Its new-usage protocol ensures non-repudiation, and therefore accountability, for inverse transparency. Its one-time pseudonym generation algorithm guarantees unlinkability and enables proof of ownership, which allows data subjects to exercise their legal rights regarding their personal data. With our implementation, we show the viability of our solution. The decentralized communication impacts performance and scalability, but exchange duration and storage size are still reasonable. More importantly, the provided information security meets high requirements. We conclude that Kovacs realizes decentralized inverse transparency through secure and GDPR-compliant use of permissionless blockchain. CCS Concepts: • Computer systems organization → Peer-to-peer architectures; • Security and privacy → Distributed systems security; Privacy-preserving protocols; Cryptography.

Hyperledger Fabric: A Distributed Operating System For Permissioned Blockchains

Abstract: Hyperledger Fabric is a modular and extensible open-source system for deploying and operating permissioned blockchains. Fabric is currently used in more than 400 prototypes and proofs-of-concept of distributed ledger technology, as well as several production systems, across different industries and use cases. Starting from the premise that there are no"one-size-fits-all"solutions, Fabric is the first truly extensible blockchain system for running distributed applications. It supports modular consensus protocols, which allows the system to be tailored to particular use cases and trust models. Fabric is also the first blockchain system that runs distributed applications written in general-purpose programming languages, without systemic dependency on a native cryptocurrency. This stands in sharp contrast to existing blockchain platforms for running smart contracts that require code to be written in domain-specific languages or rely on a cryptocurrency. Furthermore, it uses a portable notion of membership for realizing the permissioned model, which may be integrated with industry-standard identity management. To support such flexibility, Fabric takes a novel approach to the design of a permissioned blockchain and revamps the way blockchains cope with non-determinism, resource exhaustion, and performance attacks. This paper describes Fabric, its architecture, the rationale behind various design decisions, its security model and guarantees, its most prominent implementation aspects, as well as its distributed application programming model. We further evaluate Fabric by implementing and benchmarking a Bitcoin-inspired digital currency. We show that Fabric achieves end-to-end throughput of more than 3500 transactions per second in certain popular deployment configurations, with sub-second latency.


Overall, distortions produced when using Euclidean geometry for non-Euclidean geometric structures are overwhelming, causing the loss of semantic information, and hence resulting in inferior performance #REFR .

In fact, data in countless applications intrinsically has non-Euclidean geometric structures #OTHEREFR .

Several studies show that non-Euclidean geometric structures can be better captured by particular forms of Riemannian geometry #OTHEREFR .

For example, the hyperbolic geometry has a natural expressive ability for the hierarchical structure and is hence used successfully for fine-grained images #OTHEREFR .

The spherical geometry is shown as a suitable choice for face images that have the cyclical structure #OTHEREFR .

In addition to the geometric structures discussed above, natural data may be diverse and irregular in structure, e.g., data exhibits hierarchical forms in some regions and cyclical forms in others #OTHEREFR .

In this paper, we study how to attain suitable non-Euclidean geometry to capture the intrinsic geometric structures of data during continual learning.

To achieve our goal, we have to face two challenges (see Fig. 1 ).

(1) Non-stationary stream of data will inevitably increase the complexity of intrinsic geometric structures.

In other words, fixing the geometry of the underlying space cannot always match new and unseen data in continual learning.

For example, more and more complex hierarchies in a data stream bring more leaf nodes, requiring a faster growing space volume with the radius, which conflicts with a fixed geometry #OTHEREFR .

Exploring Data Geometry For Continual Learning

Abstract: Continual learning aims to efficiently learn from a nonstationary stream of data while avoiding forgetting the knowledge of old data. In many practical applications, data complies with non-Euclidean geometry. As such, the commonly used Euclidean space cannot gracefully capture non-Euclidean geometric structures of data, leading to inferior results. In this paper, we study continual learning from a novel perspective by exploring data geometry for the non-stationary stream of data. Our method dynamically expands the geometry of the underlying space to match growing geometric structures induced by new data, and prevents forgetting by keeping geometric structures of old data into account. In doing so, making use of the mixed curvature space, we propose an incremental search scheme, through which the growing geometric structures are encoded. Then, we introduce an angular-regularization loss and a neighbor-robustness loss to train the model, capable of penalizing the change of global geometric structures and local geometric structures. Experiments show that our method achieves better performance than baseline methods designed in Euclidean space.

Constant Curvature Graph Convolutional Networks

Abstract: Interest has been rising lately towards methods representing data in non-Euclidean spaces, e.g. hyperbolic or spherical, that provide specific inductive biases useful for certain real-world data properties, e.g. scale-free, hierarchical or cyclical. However, the popular graph neural networks are currently limited in modeling data only via Euclidean geometry and associated vector space operations. Here, we bridge this gap by proposing mathematically grounded generalizations of graph convolutional networks (GCN) to (products of) constant curvature spaces. We do this by i) introducing a unified formalism that can interpolate smoothly between all geometries of constant curvature, ii) leveraging gyro-barycentric coordinates that generalize the classic Euclidean concept of the center of mass. Our class of models smoothly recover their Euclidean counterparts when the curvature goes to zero from either side. Empirically, we outperform Euclidean GCNs in the tasks of node classification and distortion minimization for symbolic data exhibiting non-Euclidean behavior, according to their discrete curvature.


First, it enhances representations by incorporating strong prior knowledge from a network pre-trained on external data in the intention domain (i.e., CLINC dataset #REFR ) and adding a masked language modeling (MLM) task.

We have conducted a pilot study using the CDAC+ #OTHEREFR algorithm, which first captures pairwise sentence relationships with the guidance of labeled data and then refines cluster assignments with the DEC loss.

Another of our works, DeepAligned #OTHEREFR , initializes intent representations under the supervision of labeled data and then iteratively performs clustering and representation learning, aligning cluster centroids between adjacent iterations to obtain consistent self-supervised signals.

DCSC #OTHEREFR improves the pretraining stage by applying contrastive losses to both labeled and unlabeled data.

It mainly uses the SwAV #OTHEREFR algorithm for unsupervised learning, which requires each sample to predict the swapped view and uses Sinkhorn-Knopp #OTHEREFR to produce soft cluster assignments.

MTP-CLNN #OTHEREFR is the current state-of-the-art method, which has two key features.

Second, it adapts the SCAN algorithm #OTHEREFR to the semi-supervised setting, creating positive pairs with K-nearest neighbors or samples with the same labels for contrastive learning.

However, this method relies heavily on the selected external data, and its performance drops dramatically in a purely unsupervised scenario #OTHEREFR .

Open set recognition (OSR) #OTHEREFR and open intent detection #OTHEREFR , #OTHEREFR are two similar fields in computer vision and natural language processing, respectively.

Both fields focus on detecting unseen image-based or text-based classes during testing. However, they have limitations in distinguishing fine-grained new classes.

CD-OSR #OTHEREFR extends OSR to new class discovery, but it doesn't make use of unlabeled data during training.

Usnid: A Framework For Unsupervised And Semi-Supervised New Intent Discovery

Abstract: New intent discovery is of great value to natural language processing, allowing for a better understanding of user needs and providing friendly services. However, most existing methods struggle to capture the complicated semantics of discrete text representations when limited or no prior knowledge of labeled data is available. To tackle this problem, we propose a novel framework called USNID for unsupervised and semi-supervised new intent discovery, which has three key technologies. First, it takes full use of unsupervised or semi-supervised data to mine shallow semantic similarity relations and provide well-initialized representations for clustering. Second, it designs a centroid-guided clustering mechanism to address the issue of cluster allocation inconsistency and provide high-quality self-supervised targets for representation learning. Third, it captures high-level semantics in unsupervised or semi-supervised data to discover fine-grained intent-wise clusters by optimizing both cluster-level and instance-level objectives. We also propose an effective method for estimating the cluster number in open-world scenarios without knowing the number of new intents beforehand. USNID performs exceptionally well on several intent benchmark datasets, achieving new state-of-the-art results in unsupervised and semi-supervised new intent discovery and demonstrating robust performance with different cluster numbers. Index Terms-new intent discovery, unsupervised and semi-supervised clustering, self-supervised learning, deep neural network.

An Evaluation Dataset For Intent Classification And Out-Of-Scope Prediction

Abstract: Task-oriented dialog systems need to know when a query falls outside their range of supported intents, but current text classification corpora only define label sets that cover every example. We introduce a new dataset that includes queries that are out-of-scope---i.e., queries that do not fall into any of the system's supported intents. This poses a new challenge because models cannot assume that every query at inference time belongs to a system-supported intent class. Our dataset also covers 150 intent classes over 10 domains, capturing the breadth that a production task-oriented agent must handle. We evaluate a range of benchmark classifiers on our dataset along with several different out-of-scope identification schemes. We find that while the classifiers perform well on in-scope intent classification, they struggle to identify out-of-scope queries. Our dataset and evaluation fill an important gap in the field, offering a way of more rigorously and realistically benchmarking text classification in task-driven dialog systems.

intention domain
masked language modeling


This notion of context is the motivation behind the introduction of the attention mechanism in 2015 #REFR .

Before this, language translation was mostly relying on encoder-decoder architectures: recurrent neural networks (RNNs) #OTHEREFR and in particular long short-term memory (LSTMs) networks were used to model the relationship among words #OTHEREFR .

Specifically, each word of an input sentence is processed by the encoder sequentially.

At each step, the past and present information are summarized and encoded into a fixed-length vector.

In the end, the encoder has processed every word, and outputs a final fixed-length vector, which summarizes all input information.

This final vector is then decoded, and finally translates the input information into the target language.

Transformers And Visual Transformers

Abstract: Transformers were initially introduced for natural language processing (NLP) tasks, but fast they were adopted by most deep learning fields, including computer vision. They measure the relationships between pairs of input tokens (words in the case of text strings, parts of images for visual Transformers), termed attention. The cost is exponential with the number of tokens. For image classification, the most common Transformer Architecture uses only the Transformer Encoder in order to transform the various input tokens. However, there are also numerous other applications in which the decoder part of the traditional Transformer Architecture is also used. Here, we first introduce the Attention mechanism (Section 1), and then the Basic Transformer Block including the Vision Transformer (Section 2). Next, we discuss some improvements of visual Transformers to account for small datasets or less computation (Section 3). Finally, we introduce Visual Transformers applied to tasks other than image classification, such as detection, segmentation, generation and training without labels (Section 4) and other domains, such as video or multimodality using text or audio data (Section 5).

Neural Machine Translation By Jointly Learning To Align And Translate

Abstract: Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.


Model Training, Evaluation And Explanation

To model our dataset, we applied a standard set of regressor models like XGBoost, CatBoost, SVM, etc., and found XG-Boost #REFR to be the best-performing one.

XGBoost -a widely used algorithm for regression, classification, and ranking problems -stands for eXtreme Gradient Boosting, and it implements a gradient boosting decision trees algorithm.

XGBoost-regressor is an implementation of the XGBoost algorithm for regression problems.

It works by building a series of decision trees where each tree tries to correct the errors made by the previous tree.

In the end, the algorithm combines the results of all trees to make the final prediction.

We conducted an extensive hyperparameter search of the XGBoost regressor by experimenting with different learning rates, max depth of the tree, number of estimators to use, etc.

Using Ai To Measure Parkinson'S Disease Severity At Home

Abstract: We present an artificial intelligence system to remotely assess the motor performance of individuals with Parkinson's disease (PD). Participants performed a motor task (i.e., tapping fingers) in front of a webcam, and data from 250 global participants were rated by three expert neurologists following the Movement Disorder Society Unified Parkinson's Disease Rating Scale (MDS-UPDRS). The neurologists' ratings were highly reliable, with an intra-class correlation coefficient (ICC) of 0.88. We developed computer algorithms to obtain objective measurements that align with the MDS-UPDRS guideline and are strongly correlated with the neurologists' ratings. Our machine learning model trained on these measures outperformed an MDS-UPDRS certified rater, with a mean absolute error (MAE) of 0.59 compared to the rater's MAE of 0.79. However, the model performed slightly worse than the expert neurologists (0.53 MAE). The methodology can be replicated for similar motor tasks, providing the possibility of evaluating individuals with PD and other movement disorders remotely, objectively, and in areas with limited access to neurological care.

Xgboost: A Scalable Tree Boosting System

Abstract: Tree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.


Using the best pseudo labels from scribble prompts, DeepLab-v2 #REFR can reach 75.9% and 76.6% mIoU scores on the validation and test sets, as shown in Tab. 3.

2, we find that sampling 20% scribble pixels outperforms inputting all scribble pixels in an object by 6.4%.

Besides, iteratively inputting scribble pixels of a class can further improve the performance by 3.3%.

We analyze that iterative input is more effective for scribbles and points than image-level labels due to accurate point locations.

Finally, when inputting the scribble pixels of one class, the scribble pixels of other classes can be regarded as negative points.

We can see that adding negative points can further improve the quality of pseudo labels.

Segment Anything Is A Good Pseudo-Label Generator For Weakly Supervised Semantic Segmentation

Abstract: Weakly supervised semantic segmentation with weak labels is a long-lived illposed problem. Mainstream methods mainly focus on improving the quality of pseudo labels. In this report, we attempt to explore the potential of 'prompt to masks' from the powerful class-agnostic large segmentation model, i.e., segmentanything. Specifically, different weak labels are used as prompts to the segmentanything model, generating precise class masks. The class masks are utilized to generate pseudo labels to train the segmentation networks. We have conducted extensive experiments on PASCAL VOC 2012 dataset. Experiments demonstrate that segment-anything can serve as a good pseudo-label generator. The code will be made publicly available. * * denotes equal contribution. Preprint. Under review.

Deeplab: Semantic Image Segmentation With Deep Convolutional Nets, Atrous Convolution, And Fully Connected Crfs

Abstract: In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First , we highlight convolution with upsampled filters, or ‘atrous convolution’, as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second , we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third , we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed “DeepLab” system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7 percent mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online.


We extend the analysis of one such algorithm, the Projected Noisy Stochastic Gradient Descent (PNSGD) #REFR , restated in Algorithm 1, and establish that the algorithm satisfies (α, )-Rényi mDP with a lower noise magnitude than that required for (α, )-Rényi DP. Proposition 6.5.

Standard arguments on the composition of privacy mechanisms renders the differentially private variant of SGD (q , qδ)−differentially private at each step, where q = L/N is the lot size.

We note that, by replacing the global sensitivity assumption with input Lipschitzness, the Gaussian mechanism ensures ( L , δ)-mDP for appropriately scaled noise, as in Proposition 6.4.

Thus, DP-SGD inherently incorporates metric differential privacy, with privacy accounting based on standard composition theorems #OTHEREFR .

Metric Differential Privacy in PN-SGD When analysing the privacy of learning algorithms that require iterative updates on an intermediate solution, it is common practice to ensure privacy at each iteration and argue about the cumulative loss of privacy via composition theorems.

Another popular direction is the theoretical analysis of Noisy Stochastic Gradient Descent to formalize privacy amplifications under certain assumptions and obtain bounds on the degradation of privacy across iterations.

Let K ⊂ R d be a convex set and let {f (., x)} x∈cX be a family of convex, β-smooth functions over K, where the gradients are L input -input Lipschitz. Furthermore, assume X is a bounded set.

Then, for any η ≤ 2/β and α > 1, initializing w 0 ∈ K and dataset S ∈ X n , PNSGD run with

satisfies ( L , α) metric differential privacy.

Non-Asymptotic Lower Bounds For Training Data Reconstruction

Abstract: We investigate semantic guarantees of private learning algorithms for their resilience to training Data Reconstruction Attacks (DRAs) by informed adversaries. To this end, we derive non-asymptotic minimax lower bounds on the adversary's reconstruction error against learners that satisfy differential privacy (DP) and metric differential privacy (mDP). Furthermore, we demonstrate that our lower bound analysis for the latter also covers the high dimensional regime, wherein, the input data dimensionality may be larger than the adversary's query budget. Motivated by the theoretical improvements conferred by metric DP, we extend the privacy analysis of popular deep learning algorithms such as DP-SGD[1] and Projected Noisy SGD[2] to cover the broader notion of metric differential privacy.

Privacy Amplification By Iteration

Abstract: Many commonly used learning algorithms work by iteratively updating an intermediate solution using one or a few data points in each iteration. Analysis of differential privacy for such algorithms often involves ensuring privacy of each step and then reasoning about the cumulative privacy cost of the algorithm. This is enabled by composition theorems for differential privacy that allow releasing of all the intermediate results. In this work, we demonstrate that for contractive iterations, not releasing the intermediate results strongly amplifies the privacy guarantees. We describe several applications of this new analysis technique to solving convex optimization problems via noisy stochastic gradient descent. For example, we demonstrate that a relatively small number of non-private data points from the same distribution can be used to close the gap between private and non-private convex optimization. In addition, we demonstrate that we can achieve guarantees similar to those obtainable using the privacy-amplification-by-sampling technique in several natural settings where that technique cannot be applied.


Rq2: Can Our Classifier Learn Vulnerability In A Biased Setting?

Nevertheless, using either weighted loss or Focal Loss, the accuracy of our model drops by almost 1% compared to #REFR , indicating that these previous models are highly biased with an assumption of non-vulnerability.

However, γ is used as an exponent in Equation 7, so when we run RoBERTa-GCN w/ WL, γ is set to 0 in order to ignore its effect.

Observing the last two rows in Table 5 , we find that, initially, we tested our model using Weighted Loss (WL) only, and later we tested with Focal Loss, which is a combination of weighted loss α with the hyperparameter γ.

In both cases, we observe that our model has surpassed previous models in terms of Precision, Recall, and F1 score, indicating lower false positive and false negative rates with WL and with FL.

However, these numbers improved slightly when compared with performance between weighted loss and Focal Loss, with the exception of the precision metric.

From the results, it is observed that with Focal Loss, we achieved an improvement of 11.18% in Precision, 1.06% in Recall, and 0.61% on our F1 score compared to the weighted loss.

Compared to the next best model #OTHEREFR , our model with Focal Loss shows an improvement of 22.91% in Precision, 23.53% in Recall, and 18.04% on F1 score.

Thus, Focal Loss improves precision, recall and F1 on imbalanced data while not causing a negating impact on a balanced dataset only by adjusting the parameters α and γ RQ3: Is our classifier generalized enough to detect vulnerabilities in N-day and zero-day program sam- We evaluated our classifier's performance based on its ability to accurately predict vulnerability with 273 Nday real-world sample programs. These sample programs are never used during training.

We also used 4 zero-day examples in order to evaluate our classifier on predicting zero-day vulnerabilities as well.

The classifier predicts the vulnerability classif the vulnerability exists in the code and predicts non-vulnerable when the vulnerability does not exist.

Out of these 273 N-day and 4, zero-day code samples, some vulnerability classes exist that are not part of our VulF dataset from table 1.

An Unbiased Transformer Source Code Learning With Semantic Vulnerability Graph

Abstract: Over the years, open-source software systems have become prey to threat actors. Even highly-adopted software has been crippled by unforeseeable attacks, leaving millions of devices exposed. Even as open-source communities act quickly to patch the breach, code vulnerability screening should be an integral part of agile software development from the beginning. Unfortunately, current vulnerability screening techniques are ineffective at identifying novel vulnerabilities or providing developers with code vulnerability and classification. Furthermore, the datasets used for vulnerability learning often exhibit distribution shifts from the real-world testing distribution due to novel attack strategies deployed by adversaries and as a result, the machine learning model's performance may be hindered or biased. To address these issues, we propose a joint interpolated multitasked unbiased vulnerability classifier comprising a transformer "RoBERTa" and graph convolution neural network (GCN). We present a training process utilizing a semantic vulnerability graph (SVG) representation from source code, created by integrating edges from a sequential flow, control flow, and data flow, as well as a novel flow dubbed Poacher Flow (PF). Poacher flow edges reduce the gap between dynamic and static program analysis and handle complex long-range dependencies. Moreover, our approach reduces biases of classifiers regarding unbalanced datasets by integrating Focal Loss objective function along with SVG. Remarkably, experimental results show that our classifier outperforms state-of-the-art results on vulnerability detection with fewer false negatives and false positives. After testing our model across multiple datasets, it shows an improvement of at least 2.41% and 18.75% in the bestcase scenario. Evaluations using N-day program samples demonstrate that our proposed approach achieves a 93% accuracy and was able to detect 4, zero-day vulnerabilities from popular GitHub repositories. Our code and data are

Automated Vulnerability Detection In Source Code Using Deep Representation Learning

Abstract: Increasing numbers of software vulnerabilities are discovered every year whether they are reported publicly or discovered internally in proprietary code. These vulnerabilities can pose serious risk of exploit and result in system compromise, information leaks, or denial of service. We leveraged the wealth of C and C++ open-source code available to develop a large-scale function-level vulnerability detection system using machine learning. To supplement existing labeled vulnerability datasets, we compiled a vast dataset of millions of open-source functions and labeled it with carefully-selected findings from three different static analyzers that indicate potential exploits. The labeled dataset is available at: this https URL. Using these datasets, we developed a fast and scalable vulnerability detection tool based on deep feature representation learning that directly interprets lexed source code. We evaluated our tool on code from both real software packages and the NIST SATE IV benchmark dataset. Our results demonstrate that deep feature representation learning on source code is a promising approach for automated software vulnerability detection.


Qualitative Language Model Risk Assessment

How harms present is often highly language-dependent, and so each language needs its own dataset, but the distribution of languages represented in harm detection data is skewed #REFR .

Further, automated systems project an unknown set of values onto the result. How their creators define e.g.

"toxicity" and represent it through data is often not transparent.

Thus, not only is it hard to discover when novel forms of harm slip past undetected, it is also uncertain how well their classifications match the goal of an assessment.

Automated systems are frequently limited to well-resourced languages.

The efficacy of harm detection classifiers are limited by the amount of language-specific data.

Forms of linguistic expression evolve, but a classifier is frozen in time when it is trained (or, specifically, when its training data was gathered).

For example, some APIs would consistently mark any message containing the term "toot" as profane, causing errors first apparent when applied to Mastodon.

Automating evaluation stops assessors from learning.

A way to become better at assessing LM risks is to granularly understand their data, and output behaviours.

Assessing Language Model Deployment With Risk Cards

Abstract: This paper introduces RiskCards, a framework for structured assessment and documentation of risks associated with an application of language models. As with all language, text generated by language models can be harmful, or used to bring about harm. Automating language generation adds both an element of scale and also more subtle or emergent undesirable tendencies to the generated text. Prior work establishes a wide variety of language model harms to many different actors: existing taxonomies identify categories of harms posed by language models; benchmarks establish automated tests of these harms; and documentation standards for models, tasks and datasets encourage transparent reporting. However, there is no risk-centric framework for documenting the complexity of a landscape in which some risks are shared across models and contexts, while others are specific, and where certain conditions may be required for risks to manifest as harms. RiskCards address this methodological gap by providing a generic framework for assessing the use of a given language model in a given scenario. Each RiskCard makes clear the routes for the risk to manifest harm, their placement in harm taxonomies, and example prompt-output pairs. While RiskCards are designed to be open-source, dynamic and participatory, we present a "starter set" of RiskCards taken from a broad literature survey, each of which details a concrete risk presentation. Language model RiskCards initiate a community knowledge base which permits the mapping of risks and harms to a specific model or its application scenario, ultimately contributing to a better, safer and shared understanding of the risk landscape. CCS Concepts: • Computing methodologies → Natural language processing; • Security and privacy → Human and societal aspects of security and privacy.

Directions In Abusive Language Training Data: Garbage In, Garbage Out

Abstract: Data-driven analysis and detection of abusive online content covers many different tasks, phenomena, contexts, and methodologies. This paper systematically reviews abusive language dataset creation and content in conjunction with an open website for cataloguing abusive language data. This collection of knowledge leads to a synthesis providing evidence-based recommendations for practitioners working with this complex and highly diverse data.


C. Dual Context Network With Transformer

We remove the fifth stage and the last three pooling layers to preserve a high-resolution feature map, which has been proven to be effective in segmentation #REFR . Therefore, H 1 = H/4, W 1 = W/4.

In this section, we first introduce the architecture of the proposed DCN-T.

Since the classification results are from the derived tri-spectral image set, the final prediction can be obtained by voting. The details will be presented later.

1) Architecture: Combining the proposed DCM with an ImageNet pretrained backbone network, we obtain a complete segmentation network named DCN-T, where "T" represents the transformer used in DCM.

After reviewing the literature in the remote sensing community #OTHEREFR , it has been found that VGG-16 #OTHEREFR is one of the most frequently utilized backbones.

As a result, we adopt the VGG-16 network as the primary backbone for our study.

We add a 3 × 3 standard convolution to reduce the dimension to C = 256 in the feature F.

In the DCM, since we use one transformer encoder to obtain RACs, and then use one transformer encoder and a single-layer transformer decoder to separately encode the relationships between homogeneous areas and aggregate global contexts to form 2-D features. Therefore, N D = N E = 1.

In addition, the MLP ratio is always set to 2 for all transformer layers.

Finally, the input and output of the DCM, i.e., F and F GAC , are concatenated together.

The prediction is obtained after a convolutional layer and a bilinear upsampling layer.

Dcn-T: Dual Context Network With Transformer For Hyperspectral Image Classification

Abstract: Hyperspectral image (HSI) classification is challenging due to spatial variability caused by complex imaging conditions. Prior methods suffer from limited representation ability, as they train specially designed networks from scratch on limited annotated data. We propose a tri-spectral image generation pipeline that transforms HSI into high-quality trispectral images, enabling the use of off-the-shelf ImageNet pretrained backbone networks for feature extraction. Motivated by the observation that there are many homogeneous areas with distinguished semantic and geometric properties in HSIs, which can be used to extract useful contexts, we propose an end-to-end segmentation network named DCN-T. It adopts transformers to effectively encode regional adaptation and global aggregation spatial contexts within and between the homogeneous areas discovered by similarity-based clustering. To fully exploit the rich spectrums of the HSI, we adopt an ensemble approach where all segmentation results of the tri-spectral images are integrated into the final prediction through a voting scheme. Extensive experiments on three public benchmarks show that our proposed method outperforms state-of-the-art methods for HSI classification. The code will be released at https://github.com/DotWang/DCN-T.

Deep High-Resolution Representation Learning For Visual Recognition

Abstract: High-resolution representations are essential for position-sensitive vision problems, such as human pose estimation, semantic segmentation, and object detection. Existing state-of-the-art frameworks first encode the input image as a low-resolution representation through a subnetwork that is formed by connecting high-to-low resolution convolutions \emph{in series} (e.g., ResNet, VGGNet), and then recover the high-resolution representation from the encoded low-resolution representation. Instead, our proposed network, named as High-Resolution Network (HRNet), maintains high-resolution representations through the whole process. There are two key characteristics: (i) Connect the high-to-low resolution convolution streams \emph{in parallel}; (ii) Repeatedly exchange the information across resolutions. The benefit is that the resulting representation is semantically richer and spatially more precise. We show the superiority of the proposed HRNet in a wide range of applications, including human pose estimation, semantic segmentation, and object detection, suggesting that the HRNet is a stronger backbone for computer vision problems. All the codes are available at~{\url{this https URL}}.

segmentation
high-resolution feature map


Ablate Reference State Initialization In Dmfd

An improvement we made over the original DMfD algorithm is to disable matching with expert states, known as RSI-IR, first proposed in #REFR .

We justify this improvement in this ablation, labeled ABL5 in Fig. 7 . As shown in Fig.

10b , removing RSI and IR has a net positive effect throughout training, and around 10% on the final policy performance.

This means that matching expert states exactly via imitation reward does not help, even during the initial stages of training when the policy is randomly initialized.

We believe this is because RSI helps when there are hard-to-reach intermediate states that the policy cannot reach during the initial stages of training.

This is true for dynamic or long-horizon tasks, such as karate chops and roundhouse kicks.

Bridging Action Space Mismatch In Learning From Demonstrations

Abstract: Learning from demonstrations (LfD) methods guide learning agents to a desired solution using demonstrations from a teacher. While some LfD methods can handle small mismatches in the action spaces of the teacher and student, here we address the case where the teacher demonstrates the task in an action space that can be substantially different from that of the student-thereby inducing a large action space mismatch. We bridge this gap with a framework, Morphological Adaptation in Imitation Learning (MAIL), that allows training an agent from demonstrations by other agents with significantly different morphologies (from the student or each other). MAIL is able to learn from suboptimal demonstrations, so long as they provide some guidance towards a desired solution. We demonstrate MAIL on challenging household cloth manipulation tasks and introduce a new DRY CLOTH task-cloth manipulation in 3D task with obstacles. In these tasks, we train a visual control policy for a robot with one end-effector using demonstrations from a simulated agent with two end-effectors. MAIL shows up to 27% improvement over LfD and non-LfD baselines. It is deployed to a real Franka Panda robot, and can handle multiple variations in cloth properties (color, thickness, size, material) and pose (rotation and translation). We further show generalizability to transfers from n-to-m end-effectors, in the context of a simple rearrangement task.

Deepmimic: Example-Guided Deep Reinforcement Learning Of Physics-Based Character Skills

Abstract: A longstanding goal in character animation is to combine data-driven specification of behavior with a system that can execute a similar behavior in a physical simulation, thus enabling realistic responses to perturbations and environmental variation. We show that well-known reinforcement learning (RL) methods can be adapted to learn robust control policies capable of imitating a broad range of example motion clips, while also learning complex recoveries, adapting to changes in morphology, and accomplishing user-specified goals. Our method handles keyframed motions, highly-dynamic actions such as motion-captured flips and spins, and retargeted motions. By combining a motion-imitation objective with a task objective, we can train characters that react intelligently in interactive settings, e.g., by walking in a desired direction or throwing a ball at a user-specified target. This approach thus combines the convenience and motion quality of using motion clips to define the desired style and appearance, with the flexibility and generality afforded by RL methods and physics-based animation. We further explore a number of methods for integrating multiple clips into the learning process to develop multi-skilled agents capable of performing a rich repertoire of diverse skills. We demonstrate results using multiple characters (human, Atlas robot, bipedal dinosaur, dragon) and a large variety of skills, including locomotion, acrobatics, and martial arts.


B-MVPS #REFR achieves the best scores in 4 objects due to the usage of calibrated light information.

We use 15-view azimuth maps for optimization and leave out 5 views for testing, following PS-NeRF #OTHEREFR .

The azimuth maps are computed from the normal maps estimated by the self-calibrated photometric stereo method SDPS #OTHEREFR .

Evaluation metrics We use Chamfer distance (CD) and F-score for geometry accuracy #OTHEREFR , and mean angular error (MAE) for normal accuracy #OTHEREFR .

For CD and F-score, we only consider visible points by casting rays for all pixels and finding the first ray-mesh intersections #OTHEREFR .

Table 2 reports the geometry accuracy of the recovered DiLiGenT-MV surfaces.

UA-MVPS #OTHEREFR distorts the surface reconstruction by not considering the multi-view consistency.

MVAS outperforms PS-NeRF #OTHEREFR in 3 objects without modeling the rendering process. Figure 7 visually compares recovered "Buddha" and "Reading" objects.

Despite not having the best numerical scores, our method produces comparable results.

Lower scores for these objects are mainly due to our method's sensitivity to inaccurate silhouette masks provided by DiLiGenT-MV #OTHEREFR .

We project the GT surface onto the image plane and find up to 10-pixel inconsistency between the projected region and the GT mask. Thus, the silhouette loss Eq.

Multi-View Azimuth Stereo Via Tangent Space Consistency

Abstract: Figure 1. 3D reconstruction from calibrated multi-view azimuth maps (3 out of 31 are shown). An azimuth angle indicates the surface normal's orientation in the image plane, and an azimuth map records the azimuth angles across the entire surface. We show that azimuth maps can be effectively used for shape and normal recovery. Color images are for reference only and are not used in shape optimization.

Multi-View Photometric Stereo: A Robust Solution And Benchmark Dataset For Spatially Varying Isotropic Materials

Abstract: We present a method to capture both 3D shape and spatially varying reflectance with a multi-view photometric stereo (MVPS) technique that works for general isotropic materials. Our algorithm is suitable for perspective cameras and nearby point light sources. Our data capture setup is simple, which consists of only a digital camera, some LED lights, and an optional automatic turntable. From a single viewpoint, we use a set of photometric stereo images to identify surface points with the same distance to the camera. We collect this information from multiple viewpoints and combine it with structure-from-motion to obtain a precise reconstruction of the complete 3D shape. The spatially varying isotropic bidirectional reflectance distribution function (BRDF) is captured by simultaneously inferring a set of basis BRDFs and their mixing weights at each surface point. In experiments, we demonstrate our algorithm with two different setups: a studio setup for highest precision and a desktop setup for best usability. According to our experiments, under the studio setting, the captured shapes are accurate to 0.5 millimeters and the captured reflectance has a relative root-mean-square error (RMSE) of 9%. We also quantitatively evaluate state-of-the-art MVPS on a newly collected benchmark dataset, which is publicly available for inspiring future research.


As shown in Table 4 , UDA and ReMixMatch can still have similar robustness against backdoor attacks compared with MixMatch #REFR under our proposed ASD.

However, it can wrongly introduce a large number of poisoned samples into D C and result in the failure of our ASD, especially under WaNet and CLB.

Hence, it is necessary to control the speed to build D C and constrain the number of samples in D C during stage 1. Different semi-supervised learning methods.

We treat the samples in D P as unlabeled and apply semi-supervised learning to learn from both data pools.

In this experiment, we show our ASD can work well with various semisupervised learning, e.g., UDA #OTHEREFR and ReMixMatch #OTHEREFR .

More details about these three semisupervised learning methods are in Appendix J.

Backdoor Defense Via Adaptively Splitting Poisoned Dataset

Abstract: Backdoor defenses have been studied to alleviate the threat of deep neural networks (DNNs) being backdoor attacked and thus maliciously altered. Since DNNs usually adopt some external training data from an untrusted third party, a robust backdoor defense strategy during the training stage is of importance. We argue that the core of training-time defense is to select poisoned samples and to handle them properly. In this work, we summarize the training-time defenses from a unified framework as splitting the poisoned dataset into two data pools. Under our framework, we propose an adaptively splitting datasetbased defense (ASD). Concretely, we apply loss-guided split and meta-learning-inspired split to dynamically update two data pools. With the split clean data pool and polluted data pool, ASD successfully defends against backdoor attacks during training. Extensive experiments on multiple benchmark datasets and DNN models against six state-ofthe-art backdoor attacks demonstrate the superiority of our ASD. Our code is available at https://github.com/ KuofengGao/ASD.

Mixmatch: A Holistic Approach To Semi-Supervised Learning

Abstract: Semi-supervised learning has proven to be a powerful paradigm for leveraging unlabeled data to mitigate the reliance on large labeled datasets. In this work, we unify the current dominant approaches for semi-supervised learning to produce a new algorithm, MixMatch, that works by guessing low-entropy labels for data-augmented unlabeled examples and mixing labeled and unlabeled data using MixUp. We show that MixMatch obtains state-of-the-art results by a large margin across many datasets and labeled data amounts. For example, on CIFAR-10 with 250 labels, we reduce error rate by a factor of 4 (from 38% to 11%) and by a factor of 2 on STL-10. We also demonstrate how MixMatch can help achieve a dramatically better accuracy-privacy trade-off for differential privacy. Finally, we perform an ablation study to tease apart which components of MixMatch are most important for its success.


The advocates of linear probing #REFR argue that the probe model should be simple, e.g.

The goal of probing is to evaluate the "extractability" or "readability" of a property from the representation.

The standard approach is to train a separate model, called a probe, to predict the property p i given the fixed representations z i .

Specifically, the probing dataset D is split into a train and test set, the probe is trained on the train set, and its performance is evaluated on the test set.

Good test performance is taken as evidence that the representation contains information about the property.

Low performance indicates that the property is either not present in the representations or not usable. The idea of usability is prominent in the literature.

a logistic regression (or linear regression for continuous properties), since this means that the information can be easily extracted and used in subsequent processing (e.g. in subsequent layers). The advocates for more complex probes #OTHEREFR

(2020b) argue they are better since the information about the property may be non-linearly encoded in the representation.

Yet, the good performance of non-linear probes may come from overfitting (memorization of spurious correlations).

To overcome these limitations, various control tasks have been proposed: comparing the performance to a majority baseline, random representations, randomization of the properties, or the use of minimum description length.

Despite its limitations, in this paper we use linear probing since the results are more interpretable.

Probing Graph Representations

Abstract: Today we have a good theoretical understanding of the representational power of Graph Neural Networks (GNNs). For example, their limitations have been characterized in relation to a hierarchy of Weisfeiler-Lehman (WL) isomorphism tests. However, we do not know what is encoded in the learned representations. This is our main question. We answer it using a probing framework to quantify the amount of meaningful information captured in graph representations. Our findings on molecular datasets show the potential of probing for understanding the inductive biases of graph-based models. We compare different families of models and show that transformer-based models capture more chemically relevant information compared to models based on message passing. We also study the effect of different design choices such as skip connections and virtual nodes. We advocate for probing as a useful diagnostic tool for evaluating graph-based models.

Understanding Intermediate Layers Using Linear Classifier Probes

Abstract: Neural network models have a reputation for being black boxes. We propose a new method to better understand the roles and dynamics of the intermediate layers. This has direct consequences on the design of such models and it enables the expert to be able to justify certain heuristics (such as adding auxiliary losses in middle layers). Our method uses linear classifiers, referred to as ``probes'', where a probe can only use the hidden units of a given intermediate layer as discriminating features. Moreover, these probes cannot affect the training phase of a model, and they are generally added after training. They allow the user to visualize the state of the model at multiple steps of training. We demonstrate how this can be used to develop a better intuition about models and to diagnose potential problems.


A large dataset with approximately 129,450 images has been utilized to develop the skin cancer classification model with Inception v3 #REFR .

Computer-aided diagnosis of skin diseases has become more popular since the introduction of Inception v3 #OTHEREFR that achieved a performance accuracy of 93.3% #OTHEREFR in classifying various cancerous skin conditions.

However, gathering such a large amount of data is not feasible for some skin conditions such as Rosacea.

Although many skin conditions can lead to fatal consequences, cancer has been considered the most serious of all and has motivated the gathering of more data over time.

As a result, many Teledermatology #OTHEREFR websites have a substantial amount of skin cancer images.

On the other hand, there is very limited data for non-fatal chronic skin conditions such as Rosacea. Deep Convolutional Neural Networks (DCNNs) e.g.

Inception v3, perform relatively well provided with a large training dataset #OTHEREFR .

High Fidelity Synthetic Face Generation For Rosacea Skin Condition From Limited Data

Abstract: Similar to the majority of deep learning applications, diagnosing skin diseases using computer vision and deep learning often requires a large volume of data. However, obtaining sufficient data for particular types of facial skin conditions can be difficult due to privacy concerns. As a result, conditions like Rosacea are often understudied in computer-aided diagnosis. The limited availability of data for facial skin conditions has led to the investigation of alternative methods for computer-aided diagnosis. In recent years, Generative Adversarial Networks (GANs), mainly variants of StyleGANs, have demonstrated promising results in generating synthetic facial images. In this study, for the first time, a small dataset of Rosacea with 300 full-face images is utilized to further investigate the possibility of generating synthetic data. The preliminary experiments show how fine-tuning the model and varying experimental settings significantly affect the fidelity of the Rosacea features. It is demonstrated that R 1 Regularization strength helps achieve high-fidelity details. Additionally, this study presents qualitative evaluations of synthetic/generated faces by expert dermatologists and non-specialist participants. The quantitative evaluation is presented using a few validation metric(s). Furthermore a number of limitations and future directions are discussed. Code and generated dataset are

Rethinking The Inception Architecture For Computer Vision

Abstract: Convolutional networks are at the core of most state of-the-art computer vision solutions for a wide variety of tasks. Since 2014 very deep convolutional networks started to become mainstream, yielding substantial gains in various benchmarks. Although increased model size and computational cost tend to translate to immediate quality gains for most tasks (as long as enough labeled data is provided for training), computational efficiency and low parameter count are still enabling factors for various use cases such as mobile vision and big-data scenarios. Here we are exploring ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization. We benchmark our methods on the ILSVRC 2012 classification challenge validation set demonstrate substantial gains over the state of the art: 21:2% top-1 and 5:6% top-5 error for single frame evaluation using a network with a computational cost of 5 billion multiply-adds per inference and with using less than 25 million parameters. With an ensemble of 4 models and multi-crop evaluation, we report 3:5% top-5 error and 17:3% top-1 error on the validation set and 3:6% top-5 error on the official test set.


Secondly, the increase of interest capsules indeed improves the recall list since local businesses contain multiple aspects, while there is a saturation limit which has a similar conclusion in #REFR . Thirdly, the optimal number of is 6~8.

The lower part of Table 3 shows that M2GNNw/oL 1 sharply underperforms M2GNN without regularizing tag embeddings via L 1 , which proves the necessity of the skip-gram regularization. Impact of hyperparameters.

We investigate the influence of three hyperparameters: layer numbers, capsule numbers, and exponent before softmax, all of which greatly affect multi-interest extraction and transfer.

, max and are searched in {1, 2, 3}, {2, 4, 6, 8} and {2, 4, 6, 8}, respectively.

Figure 6 shows the results in the DPBJ dataset and it has a similar conclusion in the Amazon dataset, which is omitted. From the observations above, we make several conclusions.

Firstly, = 2 results in the best performance, which means 2-order connections integrate more similar tags into users and stacking more layers continually brings in noisy tags, which is harmful to the recommendation.

The reason is that a proper exponent helps the softmax function to identify more important interests and assign bigger scores.

M2Gnn: Metapath And Multi-Interest Aggregated Graph Neural Network For Tag-Based Cross-Domain Recommendation

Abstract: Cross-domain recommendation (CDR) is an effective way to alleviate the data sparsity problem. Content-based CDR is one of the most promising branches since most kinds of products can be described by a piece of text, especially when cold-start users or items have few interactions. However, two vital issues are still under-explored: (1) From the content modeling perspective, sufficient long-text descriptions are usually scarce in a real recommender system, more often the lightweight textual features, such as a few keywords or tags, are more accessible, which is improperly modeled by existing methods. (2) From the CDR perspective, not all inter-domain interests are helpful to infer intra-domain interests. Caused by domain-specific features, there are part of signals benefiting for recommendation in the source domain but harmful for that in the target domain. Therefore, how to distill useful interests is crucial. To tackle the above two problems, we propose a metapath and multi-interest aggregated graph neural network (M2GNN). Specifically, to model the tag-based contents, we construct a heterogeneous information network to hold the semantic relatedness between users, items, and tags in all domains. The metapath schema is predefined according to domain-specific knowledge, with one metapath for one domain. User representations are learned by GNN with a hierarchical aggregation framework, where the intra-metapath aggregation firstly filters out trivial tags and the inter-metapath aggregation further filters out useless interests. Offline experiments and online A/B tests demonstrate that M2GNN achieves significant improvements over the state-of-the-art methods and current industrial recommender system in Dianping, respectively. Further analysis shows that M2GNN offers an interpretable recommendation.

Multi-Interest Network With Dynamic Routing For Recommendation At Tmall

Abstract: Industrial recommender systems usually consist of the matching stage and the ranking stage, in order to handle the billion-scale of users and items. The matching stage retrieves candidate items relevant to user interests, while the ranking stage sorts candidate items by user interests. Thus, the most critical ability is to model and represent user interests for either stage. Most of the existing deep learning-based models represent one user as a single vector which is insufficient to capture the varying nature of user's interests. In this paper, we approach this problem from a different view, to represent one user with multiple vectors encoding the different aspects of the user's interests. We propose the Multi-Interest Network with Dynamic routing (MIND) for dealing with user's diverse interests in the matching stage. Specifically, we design a multi-interest extractor layer based on capsule routing mechanism, which is applicable for clustering historical behaviors and extracting diverse interests. Furthermore, we develop a technique named label-aware attention to help learn a user representation with multiple vectors. Through extensive experiments on several public benchmarks and one large-scale industrial dataset from Tmall, we demonstrate that MIND can achieve superior performance than state-of-the-art methods for recommendation. Currently, MIND has been deployed for handling major online traffic at the homepage on Mobile Tmall App.


B. Multi-Modal Fusion For Object Detection 1) Multi-Modal Object Detection:

Frustum PointNets #REFR extract the 3D bounding frustum of an object by extruding 2D bounding boxes from image detectors.

To date, several studies have investigated multi-modal fusion for 2D and 3D object detection.

PointFusion #OTHEREFR combines a CNN and a PointNet #OTHEREFR architecture respectively to process images and raw point clouds then predict 3D boxes.

PointPainting #OTHEREFR projects LiDAR points into the output of an image-only semantic segmentation network, and appends the class scores to each point.

All these fusion methods of RGB and LiDAR achieve high average precision on the benchmarks, however, the coupling or interrelation of two modalities will cause the whole system to fail easily once part of the sensors break down.

Besides, the methods above only provide a deterministic predict result, making it risky to carry out in the real application.

2) Adaptive fusion: Several new studies have proposed self-adaptive techniques in computer vision.

Informative Data Selection With Uncertainty For Multi-Modal Object Detection

Abstract: Noise has always been nonnegligible trouble in object detection by creating confusion in model reasoning, thereby reducing the informativeness of the data. It can lead to inaccurate recognition due to the shift in the observed pattern, that requires a robust generalization of the models. To implement a general vision model, we need to develop deep learning models that can adaptively select valid information from multi-modal data. This is mainly based on two reasons. Multi-modal learning can break through the inherent defects of single-modal data, and adaptive information selection can reduce chaos in multi-modal data. To tackle this problem, we propose a universal uncertainty-aware multi-modal fusion model. It adopts a multi-pipeline loosely coupled architecture to combine the features and results from point clouds and images. To quantify the correlation in multimodal information, we model the uncertainty, as the inverse of data information, in different modalities and embed it in the bounding box generation. In this way, our model reduces the randomness in fusion and generates reliable output. Moreover, we conducted a completed investigation on the KITTI 2D object detection dataset and its derived dirty data. Our fusion model is proven to resist severe noise interference like Gaussian, motion blur, and frost, with only slight degradation. The experiment results demonstrate the benefits of our adaptive fusion. Our analysis on the robustness of multi-modal fusion will provide further insights for future research.

Frustum Pointnets For 3D Object Detection From Rgb-D Data

Abstract: In this work, we study 3D object detection from RGB-D data in both indoor and outdoor scenes. While previous methods focus on images or 3D voxels, often obscuring natural 3D patterns and invariances of 3D data, we directly operate on raw point clouds by popping up RGB-D scans. However, a key challenge of this approach is how to efficiently localize objects in point clouds of large-scale scenes (region proposal). Instead of solely relying on 3D proposals, our method leverages both mature 2D object detectors and advanced 3D deep learning for object localization, achieving efficiency as well as high recall for even small objects. Benefited from learning directly in raw point clouds, our method is also able to precisely estimate 3D bounding boxes even under strong occlusion or with very sparse points. Evaluated on KITTI and SUN RGB-D 3D detection benchmarks, our method outperforms the state of the art by remarkable margins while having real-time capability.


In general, there are two different interpretations: intelligence as either a collection of task-specific skills or a general learning ability #REFR ], which reflect the distinction between crystallised and fluid abilities, respectively.

Intelligence is a controversial and elusive concept subject to philosophical debate #OTHEREFR

2007] , best understood as a nomological network of constructs #OTHEREFR .

Etymologically, intelligence comes from Latin "intelligere", which means "to understand".

It can be defined as "the global capacity of the individual to act purposefully, to think rationally, and to deal effectively with the environment" #OTHEREFR ], or the property that "measures an agent's ability to achieve goals in a wide range of environments" #OTHEREFR .

Problems about intelligence include, for instance, its definition and modelling, such as devising the structure of intelligence #OTHEREFR

2011] ; its relation with action; its measurement and evaluation; its analysis; and its construction and development.

Concerning the theories of intelligence, there are two main traditions #OTHEREFR

2011] : the psychometric tradition, based on the number and nature of basic cognitive abilities or factors; and the developmental or holistic perspective, based on acquired intellect.

The problem of the measure of intelligence #OTHEREFR is of course related to what representation or model of intelligence is considered, and is complicated by the need of distinguishing between causality and correlation, selecting a representative set of environments for evaluation, etc. Carrol defines an ability (i.e.

Artificial Collective Intelligence Engineering: A Survey Of Concepts And Perspectives

Abstract: Collectiveness is an important property of many systems-both natural and artificial. By exploiting a large number of individuals, it is often possible to produce effects that go far beyond the capabilities of the smartest individuals, or even to produce intelligent collective behaviour out of not-so-intelligent individuals. Indeed, collective intelligence, namely the capability of a group to act collectively in a seemingly intelligent way, is increasingly often a design goal of engineered computational systems-motivated by recent techno-scientific trends like the Internet of Things, swarm robotics, and crowd computing, just to name a few. For several years, the collective intelligence observed in natural and artificial systems has served as a source of inspiration for engineering ideas, models, and mechanisms. Today, artificial and computational collective intelligence are recognised research topics, spanning various techniques, kinds of target systems, and application domains. However, there is still a lot of fragmentation in the research panorama of the topic within computer science, and the verticality of most communities and contributions makes it difficult to extract the core underlying ideas and frames of reference. The challenge is to identify, place in a common structure, and ultimately connect the different areas and methods addressing intelligent collectives. To address this gap, this paper considers a set of broad scoping questions providing a map of collective intelligence research, mostly by the point of view of computer scientists and engineers. Accordingly, it covers preliminary notions, fundamental concepts, and the main research perspectives, identifying opportunities and challenges for researchers on artificial and computational collective intelligence engineering.

On The Measure Of Intelligence

Abstract: To make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans. Over the past hundred years, there has been an abundance of attempts to define and measure intelligence, across both the fields of psychology and AI. We summarize and critically assess these definitions and evaluation approaches, while making apparent the two historical conceptions of intelligence that have implicitly guided them. We note that in practice, the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs and humans at specific tasks such as board games and video games. We argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to "buy" arbitrary levels of skills for a system, in a way that masks the system's own generalization power. We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. Using this definition, we propose a set of guidelines for what a general AI benchmark should look like. Finally, we present a benchmark closely following these guidelines, the Abstraction and Reasoning Corpus (ARC), built upon an explicit set of priors designed to be as close as possible to innate human priors. We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.


We note that the syndrome does not depend on the codeword and, therefore, we do not require the NN to have a special structure, it can be arbitrary, but the best results were obtained with recurrent NNs #REFR .

The next idea was to replace the activation functions, the architecture is called a hyper-network #OTHEREFR , #OTHEREFR . Later, Cammerer et al.

proposed to replace node and edge message updates with trainable functions, thus allowing NN to learn a generalized message passing algorithm #OTHEREFR .

Another approach proposed in #OTHEREFR is to consider the syndrome-based decoding algorithm that is suitable for any linear codes.

The basic syndrome-based decoding algorithm implies the use of the mapping (syndrome to the coset leader), which has the exponential (in the number of parity-check bits) size.

The idea of #OTHEREFR is to approximate this table with a NN.

Later a syndrome-based approach was adapted to the transformer and denoising diffusion architectures #OTHEREFR , #OTHEREFR .

We also note the papers (see, e.g. #OTHEREFR ) devoted to DNN-based code construction.

For additional literature and a more detailed overview, we refer the reader to #OTHEREFR .

The papers above focus on the performance of hard-output decoding, i.e.

the decoder is required to return the estimated information word.

Soft-Output Deep Neural Network-Based Decoding

Abstract: Deep neural network (DNN)-based channel decoding is widely considered in the literature. The existing solutions are investigated for the case of hard output, i.e. when the decoder returns the estimated information word. At the same time, soft-output decoding is of critical importance for iterative receivers and decoders. In this paper, we focus on the soft-output DNN-based decoding problem. We start with the syndrome-based approach proposed by Bennatan et al. (2018) and modify it to provide soft output in the AWGN channel. The new decoder can be considered as an approximation of the MAP decoder with smaller computation complexity. We discuss various regularization functions for joint DNN-MAP training and compare the resulting distributions for [64, 45] BCH code. Finally, to demonstrate the soft-output quality we consider the turboproduct code with [64, 45] BCH codes as row and column codes. We show that the resulting DNN-based scheme is very close to the MAP-based performance and significantly outperforms the solution based on the Chase decoder. We come to the conclusion that the new method is prospective for the challenging problem of DNN-based decoding of long codes consisting of short component codes.

Deep Learning For Decoding Of Linear Codes - A Syndrome-Based Approach

Abstract: We present a novel framework for applying deep neural networks (DNN) to soft decoding of linear codes at arbitrary block lengths. Unlike other approaches, our framework allows unconstrained DNN design, enabling the free application of powerful designs that were developed in other contexts. Our method is robust to overfitting that inhibits many competing methods, which follows from the exponentially large number of codewords required for their training. We achieve this by transforming the channel output before feeding it to the network, extracting only the syndrome of the hard decisions and the channel output reliabilities. We prove analytically that this approach does not involve any intrinsic performance penalty, and guarantees the generalization of performance obtained during training. Our best results are obtained using a recurrent neural network (RNN) architecture combined with simple preprocessing by permutation. We provide simulation results that demonstrate performance that sometimes approaches that of the ordered statistics decoding (OSD) algorithm.


If c 1 ≥ 2, then d c1 ≤ 1 2 , i.e., for size 1, the solution has value d c1 ≤ #REFR 2 which implies that the solution has a competitive ratio of at least 2. Thus assume that c 1 = 1.

There are even discrete instances where every continuization of the instance has a competitive ratio smaller than the initial instance.

For i ∈ {2, 5, 6, 7, 8, 9, 10, 11}, the density is chosen such that i

We show that every incremental solution (c 1 , c 2 , . . .

) has a competitive ratio of at least 1.446 for this problem instance.

If c 2 ≥ 5, we can, without loss of generality, assume that c 2 ≥ 12.

Otherwise we can improve the solution by choosing c 2 = 4 instead.

Then, the value of the solution for size 4 is max{1, 3

107200 } = 1, while the optimal solution has value 4d 4 = 17 10 , i.e., the competitive ratio of the solution is at least 1.7.

Incremental Maximization Via Continuization

Abstract: We consider the problem of finding an incremental solution to a cardinality-constrained maximization problem that not only captures the solution for a fixed cardinality, but also describes how to gradually grow the solution as the cardinality bound increases. The goal is to find an incremental solution that guarantees a good competitive ratio against the optimum solution for all cardinalities simultaneously. The central challenge is to characterize maximization problems where this is possible, and to determine the best-possible competitive ratio that can be attained. A lower bound of 2.18 and an upper bound of ϕ+1 ≈ 2.618 are known on the competitive ratio for monotone and accountable objectives [Bernstein et al., Math. Prog., 2022], which capture a wide range of maximization problems. We introduce a continuization technique and identify an optimal incremental algorithm that provides strong evidence that ϕ + 1 is the best-possible competitive ratio. Using this continuization, we obtain an improved lower bound of 2.246 by studying a particular recurrence relation whose characteristic polynomial has complex roots exactly beyond the lower bound. Based on the optimal continuous algorithm combined with a scaling approach, we also provide a 1.772-competitive randomized algorithm. We complement this by a randomized lower bound of 1.447 via Yao's principle.

General Bounds For Incremental Maximization

Abstract: We propose a theoretical framework to capture incremental solutions to cardinality constrained maximization problems. The defining characteristic of our framework is that the cardinality/support of the solution is bounded by a value $k\in\mathbb{N}$ that grows over time, and we allow the solution to be extended one element at a time. We investigate the best-possible competitive ratio of such an incremental solution, i.e., the worst ratio over all $k$ between the incremental solution after $k$ steps and an optimum solution of cardinality $k$. We define a large class of problems that contains many important cardinality constrained maximization problems like maximum matching, knapsack, and packing/covering problems. We provide a general $2.618$-competitive incremental algorithm for this class of problems, and show that no algorithm can have competitive ratio below $2.18$ in general. In the second part of the paper, we focus on the inherently incremental greedy algorithm that increases the objective value as much as possible in each step. This algorithm is known to be $1.58$-competitive for submodular objective functions, but it has unbounded competitive ratio for the class of incremental problems mentioned above. We define a relaxed submodularity condition for the objective function, capturing problems like maximum (weighted) ($b$-)matching and a variant of the maximum flow problem. We show that the greedy algorithm has competitive ratio (exactly) $2.313$ for the class of problems that satisfy this relaxed submodularity condition. Note that our upper bounds on the competitive ratios translate to approximation ratios for the underlying cardinality constrained problems.


We first train a U-Net #REFR on GAMMA and then evaluate on the testing set of the degraded REFUGE dataset. VSD.

3 , higher values of PSNR or SSIM do not necessarily indicate better preservation of structural details, as already pointed out elsewhere #OTHEREFR .

To assess the fundus image enhancement performance with respect to structural details and clinically-relevant applications, we propose the following evaluation metrics: FIQA.

Similar to #OTHEREFR , we train a ResNet101 #OTHEREFR on EyeQ for quality assessment and use the predictions to calculate a fundus image quality assessment score (FIQA).

We evaluate on the original testing set of EyeQ labeled as "Usable". OCSD.

We use the OC segmentation Dice (OCSD) as a metric of assessing the capability of preserving anatomical structures.

We use the vessel segmentation Dice (VSD) as another metric of assessing the capability of preserving anatomical structures.

Iter-Net #OTHEREFR is used to segment vessels on the testing set of the degraded DRIVE dataset. FCNR.

Inspired by the contrast noise ratio #OTHEREFR , we propose a fundus contrast noise ratio (FCNR) to evaluate the contrast quality of the vessel areas.

We first obtain a region of interest (ROI) R using a disk-shaped dilation operation with a radius of 3 pixels based on the vessel segmentation mask.

We formulate the vessel area as V while the background area as B. The FCNR is defined as

Learning Enhancement From Degradation: A Diffusion Model For Fundus Image Enhancement

Abstract: The quality of a fundus image can be compromised by numerous factors, many of which are challenging to be appropriately and mathematically modeled. In this paper, we introduce a novel diffusion model based framework, named Learning Enhancement from Degradation (LED), for enhancing fundus images. Specifically, we first adopt a data-driven degradation framework to learn degradation mappings from unpaired high-quality to low-quality images. We then apply a conditional diffusion model to learn the inverse enhancement process in a paired manner. The proposed LED is able to output enhancement results that maintain clinically important features with better clarity. Moreover, in the inference phase, LED can be easily and effectively integrated with any existing fundus image enhancement framework. We evaluate the proposed LED on several downstream tasks with respect to various clinically-relevant metrics, successfully demonstrating its superiority over existing state-of-the-art methods both quantitatively and qualitatively.

U-Net: Convolutional Networks For Biomedical Image Segmentation

Abstract: There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .


We find that the baseline already performs competitively on our benchmarks, despite the relatively small size of SMiT compared to other datasets such as HowTo100M #REFR , due to the quality and diversity of the manually annotated captions.

Spoken Moments in Time (SMiT) pretraining dataset.

The SMiT #OTHEREFR training set consists of 481K pairs of 3 seconds video clips with corresponding captions.

It is a subset of Moments in Time (MiT) #OTHEREFR .

Our work falls under the umbrella of transfer learning: we pretrain on SMiT and then use the resulting features to solve different downstream tasks in a zero-shot or fine-tuned manner.

Pretraining is either done as in regular contrastive learning ('baseline') or with our VFC framework.

We encourage the community to consider SMiT as a powerful pretraining dataset. PaLM.

We use PaLM-540B #OTHEREFR with beam size 4, output sequence length 512, and temperature of 0.7.

The negative captions are generated in an autogressive way and are therefore of arbitrary length.

We post-process them by removing text after any newline character and by filtering out candidates which contain the same verbs as the original caption.

Training details. Most hyper-parameters follow CLIP4CLIP #OTHEREFR .

Verbs In Action: Improving Verb Understanding In Video-Language Models

Abstract: two brown horses eating grass LLM Original caption Video Add prompt two brown horses running on the grass two brown horses fighting on the grass two brown horses lying on the grass eating grass Verb phrase Hard verb negative captions two brown horses sleeping on the grass two brown horses playing on the grass eating grass cleaning camera standing squatting Verb phrase loss two brown horses eating grass two brown horses lying on the grass a woman standing in a post office person squatting at the gym a man cleaning his camera Hard negative loss Generated hard negative caption Negative captions in the batch Negative verb phrases in the batch Batch Figure 1. Verb-Focused Contrastive (VFC) learning: (Left): Given a video and its corresponding caption, we leverage a Large Language Model (LLM) to output (1) hard negative captions, where only the verb has been changed while keeping the remaining context, and (2) verb phrases which succinctly describe the action in the video. (Right): To encourage better verb reasoning, we subsequently enforce (1) a calibrated hard negative loss, using our generated hard negative captions and other captions in the batch, and (2) a fine-grained, verb phrase loss. We show that VFC improves verb understanding of video-language models compared to the standard contrastive loss.

Howto100M: Learning A Text-Video Embedding By Watching Hundred Million Narrated Video Clips

Abstract: Learning text-video embeddings usually requires a dataset of video clips with manually provided captions. However, such datasets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations. The contributions of this work are three-fold. First, we introduce HowTo100M: a large-scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks. Our data collection procedure is fast, scalable and does not require any additional manual annotation. Second, we demonstrate that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask. Finally, we show that this embedding transfers well to other domains: fine-tuning on generic Youtube videos (MSR-VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone. Our dataset, code and models will be publicly available at: www.di.ens.fr/willow/research/howto100m/.