In Pix2Pix, using instance normalization, which involves setting the batch size to 1, is a better choice than batch normalization for training generative models #REFR .

One benefit of the PatchGAN discriminator is its ability to apply a fixed-size patch discriminator on images of any size, unlike typical discriminators that only score the full image.

Additionally, a smaller PatchGAN has fewer parameters, runs faster, and can be applied to images of any size.

Training and optimization: GAN models are notoriously challenging to train and converge due to the simultaneous training of the generator and discriminator, with improvements to one model often coming at the expense of the other.

The differing learning rates of the two models can also make convergence difficult.

As suggested in the original GAN paper #OTHEREFR , instead of training the generator G to minimize log(1 − D(x, G(x, z))), it is more effective to train G to maximize log(D(x, G(x, z))) to improve convergence.

Exploring The Power Of Generative Deep Learning For Image-To-Image Translation And Mri Reconstruction: A Cross-Domain Review

Abstract: Deep learning has become a prominent computational modeling tool in the areas of computer vision and image processing in recent years. This research comprehensively analyzes the different deep-learning methods used for image-to-image translation and reconstruction in the natural and medical imaging domains. We examine the famous deep learning frameworks, such as convolutional neural networks and generative adversarial networks, and their variants, delving into the fundamental principles and difficulties of each. In the field of natural computer vision, we investigate the development and extension of various deeplearning generative models. In comparison, we investigate the possible applications of deep learning to generative medical imaging problems, including medical image translation, MRI reconstruction, and multi-contrast MRI synthesis. This thorough review provides scholars and practitioners in the areas of generative computer vision and medical imaging with useful insights for summarizing past works and getting insight into future research paths.

Instance Normalization: The Missing Ingredient For Fast Stylization

Abstract: It this paper we revisit the fast stylization method introduced in Ulyanov et. al. (2016). We show how a small change in the stylization architecture results in a significant qualitative improvement in the generated images. The change is limited to swapping batch normalization with instance normalization, and to apply the latter both at training and testing times. The resulting method can be used to train high-performance architectures for real-time image generation. The code will is made available on github at this https URL. Full paper can be found at arXiv:1701.02096.

generative models
instance normalization


Hotspot Identification Via Fish Detection

Next, we also investigate using a fully out-of-sample detector, which we call the MegaFishDetector (MFD), similar to #REFR , but trained using several publicly available fish detection datasets using the same YOLOv5m at 1280p network.

However, for these detectors to be useful and drastically reduce annotation effort, they must detect fish with similar accuracy on other reefs, or at different times in various conditions.

We thus provide an initial investigation by deploying the same detector used for the in-sample study, and apply it without modification to a survey on another coral We see that MFD alone is insufficient, and JS alone is still not robust even to small changes in site context.

JSMF D is slightly more robust to out of sample data, but still more work needs to be done.

We minimize changes in the environment by selecting a reef that is nearby to the original, and survey at a similar time of day and weather conditions, only one day apart.

This dataset consists of roughly 210K training images and 36K validation images, assembled together from the following datasets: AIMS OzFish #OTHEREFR , FathomNet (only the gnathostomata class) #OTHEREFR , VIAME FishTrack22 (subsampled by 50%) #OTHEREFR , NOAA Puget Sound Nearshore Fish #OTHEREFR , NOAA Labeled Fishes In the Wild #OTHEREFR , and DeepFish (Segmentation only) #OTHEREFR .

Finally, we compare the ability of using MFD alone or as a pre-trained network that can be used to fine-tune the in-sample network.

MFD code and models are available at https:// github.com/warplab/megafishdetector.

Biological Hotspot Mapping In Coral Reefs With Robotic Visual Surveys

Abstract: Figure 1. Hotspot detection at two unique coral reef sites in the US Virgin Islands-Joel's Shoal (JS) and Tektite (TK). From visual surveys collected by an AUV, we generate hotspot maps of fish counts along the robot trajectory and plotted with a log-colorscale (a/d). Our fish detector is fine-tuned on labelled images from JS, but not from TK. We can see distinct hotspots at both reefs, correlated with distinct reef structure visible in our 3D reconstruction, an overhanging structure and a pillar coral (b), as well as with rugosity (c), which is a measure of topographic complexity used by the biological community as a proxy for biodiversity and abundance. Our system enables rapid deployment of an AUV over a predefined but unknown area to identify biologically interesting hotspots.

Efficient Pipeline For Camera Trap Image Review

Abstract: Biologists all over the world use camera traps to monitor biodiversity and wildlife population density. The computer vision community has been making strides towards automating the species classification challenge in camera traps, but it has proven difficult to to apply models trained in one region to images collected in different geographic areas. In some cases, accuracy falls off catastrophically in new region, due to both changes in background and the presence of previously-unseen species. We propose a pipeline that takes advantage of a pre-trained general animal detector and a smaller set of labeled images to train a classification model that can efficiently achieve accurate results in a new region.


A deep residual learning framework (ResNet) is proposed to overcome the degradation issue by #REFR .

In this work, there are four different networks applied as the backbone of DINO, DINO-TP, and DINO-MC, including ViT, Swin Transformer, ResNet, and WRN.

ViT preserves the Transformer structure used in NLP as much as possible and performs very well in computer vision tasks #OTHEREFR .

The input of transformer encoder is the embedding formed by adding the patch embedding with the position encoding.

The transformer encoder of ViT consists mainly of layer normalization (LN) which is applied before each block, multi-head attention, and multi-layer perceptron block (MLP).

Inspired by ViT, Swin Transformer is proposed with a hierarchical transformer, which computes representation with both regular and shifted windows to capture features from different levels and resolutions.

They add shortcut connections to transmit the information of shallow layers directly to the deeper layers of the neural network and require no additional computation.

With this mechanism, the residual net is deepened to 152 layers and achieves stateof-the-art results in multiple computer vision tasks.

Furthermore, with the development of self-supervised representation learning, ResNet also becomes an effective back-bone widely used in multiple self-supervised architectures #OTHEREFR .

WRN [49] is proposed to improve the performance of ResNet by increase the width of the residual block, i.e., widening the convolutional layers.

When the same number of parameters are utilized, wide residual block can get superior outcomes with less training time compared to the original residual block.

Dino-Mc: Self-Supervised Contrastive Learning For Remote Sensing Imagery With Multi-Sized Local Crops

Abstract: Due to the costly nature of remote sensing image labeling and the large volume of available unlabeled imagery, self-supervised methods that can learn feature representations without manual annotation have received great attention. While prior works have explored self-supervised learning in remote sensing tasks, pretext tasks based on local-global view alignment remain underexplored. Inspired by DINO [6], which employs an effective representation learning structure with knowledge distillation based on global-local view alignment, we formulate two pretext tasks for use in self-supervised learning on remote sensing imagery (SSLRS). Using these tasks, we explore the effectiveness of positive temporal contrast as well as multi-sized views on SSLRS. Moreover, we extend DINO and propose DINO-MC which uses local views of various sized crops instead of a single fixed size. Our experiments demonstrate that even when pre-trained on only 10% of the dataset, DINO-MC performs on par or better than existing state of the art SSLRS methods on multiple remote sensing tasks, while using less computational resources. All codes, models and results are available at https://github.com/ WennyXY/DINO-MC.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


Moreover, this discretization depends on the chosen bin size and does not track the mutual information across varying bin sizes #REFR ; Ross (2014).

Information-theoretic methods have made a significant impact on deep learning #OTHEREFR .

However, a key challenge is addressing the source of randomness in deterministic DNNs.

The mutual information between the input and representation is infinite, leading to ill-posed optimization problems or piecewise constant outcomes #OTHEREFR . To tackle this issue, researchers have proposed various solutions.

One common approach is to discretize the input distribution and real-valued hidden representations by binning, which facilitates non-trivial measurements and prevents the mutual information from always taking the maximum value of the log of the dataset size, thus avoiding ill-posed optimization problems #OTHEREFR .

However, binning and discretization are essentially equivalent to geometrical compression and serve as clustering measures #OTHEREFR .

To address these limitations, researchers have proposed alternative approaches such as interpreting binned information as a weight decay penalty #OTHEREFR

(2019b) , estimating mutual information based on lower bounds assuming a continuous input distribution without making assumptions about the network's output distribution properties #OTHEREFR , injecting additive noise, and considering data augmentation as the source of noise (Dubois et al., 2021; #OTHEREFR .

To Compress Or Not To Compress -Self-Supervised Learning And Information Theory: A Review

Abstract: Deep neural networks have demonstrated remarkable performance in supervised learning tasks but require large amounts of labeled data. Self-supervised learning offers an alternative paradigm, enabling the model to learn from data without explicit labels. Information theory has been instrumental in understanding and optimizing deep neural networks. Specifically, the information bottleneck principle has been applied to optimize the trade-off between compression and relevant information preservation in supervised settings. However, the optimal information objective in self-supervised learning remains unclear. In this paper, we review various approaches to self-supervised learning from an information-theoretic standpoint and present a unified framework that formalizes the self-supervised informationtheoretic learning problem. We integrate existing research into a coherent framework, examine recent self-supervised methods, and identify research opportunities and challenges. Moreover, we discuss empirical measurement of information-theoretic quantities and their estimators. This paper offers a comprehensive review of the intersection between information theory, self-supervised learning, and deep neural networks.

Estimating Information Flow In Neural Networks

Abstract: We study the flow of information and the evolution of internal representations during deep neural network (DNN) training, aiming to demystify the compression aspect of the information bottleneck theory. The theory suggests that DNN training comprises a rapid fitting phase followed by a slower compression phase, in which the mutual information $I(X;T)$ between the input $X$ and internal representations $T$ decreases. Several papers observe compression of estimated mutual information on different DNN models, but the true $I(X;T)$ over these networks is provably either constant (discrete $X$) or infinite (continuous $X$). This work explains the discrepancy between theory and experiments, and clarifies what was actually measured by these past works. To this end, we introduce an auxiliary (noisy) DNN framework for which $I(X;T)$ is a meaningful quantity that depends on the network's parameters. This noisy framework is shown to be a good proxy for the original (deterministic) DNN both in terms of performance and the learned representations. We then develop a rigorous estimator for $I(X;T)$ in noisy DNNs and observe compression in various models. By relating $I(X;T)$ in the noisy DNN to an information-theoretic communication problem, we show that compression is driven by the progressive clustering of hidden representations of inputs from the same class. Several methods to directly monitor clustering of hidden representations, both in noisy and deterministic DNNs, are used to show that meaningful clusters form in the $T$ space. Finally, we return to the estimator of $I(X;T)$ employed in past works, and demonstrate that while it fails to capture the true (vacuous) mutual information, it does serve as a measure for clustering. This clarifies the past observations of compression and isolates the geometric clustering of hidden representations as the true phenomenon of interest.


These results corroborate the "Rashomon" theory #REFR that, for some problems, there exist simple models with good accuracy and meet special requirements, such as the limited memory size of embedded devices in this case.

Note that, 300-second HRV monitoring can be used for caring for chronic renal failure and diabetes #OTHEREFR , showing the healthcare potential of PPG-based HRV monitoring.

The evaluation results also show that our method is significantly more accurate than the signal-processing-only and ML-only methods.

Moreover, the model exploration showed that DT and MLP models are usually smaller with good accuracy, making them more suitable for small embedded devices.

DT models have an average error of 13.0% and are usually less than 10KB with inference time less than 10 s.

MLP models have an average error of 9.1% and are less than 469KB with inference time less than 1ms.

The contributions of this paper include,

• The compound and direct HRV inference methodology combines signal processing and ML to directly infer RMSSD/SDNN to achieve high accuracy with small and fast ML models. Figure 1 .

An example of ECG waveform showcasing R peaks (red dots) and RR intervals.

• A systematic exploration of different ML algorithms to study their impact on the accuracy, model size, and time on HRV inference.

Efficient And Direct Inference Of Heart Rate Variability Using Both Signal Processing And Machine Learning

Abstract: Heart Rate Variability (HRV) measures the variation of the time between consecutive heartbeats and is a major indicator of physical and mental health. Recent research has demonstrated that photoplethysmography (PPG) sensors can be used to infer HRV. However, many prior studies had high errors because they only employed signal processing or machine learning (ML), or because they indirectly inferred HRV, or because there lacks large training datasets. Many prior studies may also require large ML models. The low accuracy and large model sizes limit their applications to small embedded devices and potential future use in healthcare. To address the above issues, we first collected a large dataset of PPG signals and HRV ground truth. With this dataset, we developed HRV models that combine signal processing and ML to directly infer HRV. Evaluation results show that our method had errors between 3.5% to 25.7% and outperformed signal-processing-only and ML-only methods. We also explored different ML models, which showed that Decision Trees and Multi-level Perceptrons have 13.0% and 9.1% errors on average with models at most hundreds of KB and inference time less than 1ms. Hence, they are more suitable for small embedded devices and potentially enable the future use of PPG-based HRV monitoring in healthcare.

A Study In Rashomon Curves And Volumes: A New Perspective On Generalization And Model Simplicity In Machine Learning

Abstract: The Rashomon effect occurs when many different explanations exist for the same phenomenon. In machine learning, Leo Breiman used this term to characterize problems where many accurate-but-different models exist to describe the same data. In this work, we study how the Rashomon effect can be useful for understanding the relationship between training and test performance, and the possibility that simple-yet-accurate models exist for many problems. We consider the Rashomon set - the set of almost-equally-accurate models for a given problem - and study its properties and the types of models it could contain. We present the Rashomon ratio as a new measure related to simplicity of model classes, which is the ratio of the volume of the set of accurate models to the volume of the hypothesis space; the Rashomon ratio is different from standard complexity measures from statistical learning theory. For a hierarchy of hypothesis spaces, the Rashomon ratio can help modelers to navigate the trade-off between simplicity and accuracy. In particular, we find empirically that a plot of empirical risk vs. Rashomon ratio forms a characteristic $\Gamma$-shaped Rashomon curve, whose elbow seems to be a reliable model selection criterion. When the Rashomon set is large, models that are accurate - but that also have various other useful properties - can often be obtained. These models might obey various constraints such as interpretability, fairness, or monotonicity.


For example, if we apply HGPF on HAN #REFR , the running time of HGPF on ACM dataset is about 5 minutes with a single GPU device of GeForce RTX 3090.

The time and space complexity of our framework is linear to the scale of an HIN, i.e., the number of nodes, edges, features and meta-path instances.

In fact, the training of HGPF is very time efficient.

A Post-Training Framework For Improving Heterogeneous Graph Neural Networks

Abstract: Recent years have witnessed the success of heterogeneous graph neural networks (HGNNs) in modeling heterogeneous information networks (HINs). In this paper, we focus on the benchmark task of HGNNs, i.e., node classification, and empirically find that typical HGNNs are not good at predicting the label of a test node whose receptive field (1) has few training nodes from the same category or (2) has multiple training nodes from different categories. A possible explanation is that their message passing mechanisms may involve noises from different categories, and cannot fully explore task-specific knowledge such as the label dependency between distant nodes. Therefore, instead of introducing a new HGNN model, we propose a general post-training framework that can be applied on any pretrained HGNNs to further inject task-specific knowledge and enhance their prediction performance. Specifically, we first design an auxiliary system that estimates node labels based on (1) a global inference module of multi-channel label propagation and (2) a local inference module of network schema-aware prediction. The mechanism of our auxiliary system can complement the pretrained HGNNs by providing extra task-specific knowledge. During the post-training process, we will strengthen both system-level and module-level consistencies to encourage the cooperation between a pretrained HGNN and our auxiliary system. In this way, both systems can learn from each other for better performance. In experiments, we apply our framework to four typical HGNNs. Experimental results on three benchmark datasets show that compared with pretrained HGNNs, our post-training framework can enhance Micro-F1 by a relative improvement of 3.9% on average. Code, data

Heterogeneous Graph Attention Network

Abstract: Graph neural network, as a powerful graph representation technique based on deep learning, has shown superior performance and attracted considerable research interest. However, it has not been fully considered in graph neural network for heterogeneous graph which contains different types of nodes and links. The heterogeneity and rich semantic information bring great challenges for designing a graph neural network for heterogeneous graph. Recently, one of the most exciting advancements in deep learning is the attention mechanism, whose great potential has been well demonstrated in various areas. In this paper, we first propose a novel heterogeneous graph neural network based on the hierarchical attention, including node-level and semantic-level attentions. Specifically, the node-level attention aims to learn the importance between a node and its metapath based neighbors, while the semantic-level attention is able to learn the importance of different meta-paths. With the learned importance from both node-level and semantic-level attention, the importance of node and meta-path can be fully considered. Then the proposed model can generate node embedding by aggregating features from meta-path based neighbors in a hierarchical manner. Extensive experimental results on three real-world heterogeneous graphs not only show the superior performance of our proposed model over the state-of-the-arts, but also demonstrate its potentially good interpretability for graph analysis.


SSL with Transformers has driven recent advances in NLP and computer vision (e.g. #REFR ; Dosovitskiy et al. (2021)), and has clear potential to advance other fields.

(2022) uses Transformers instead in an alternating manner, performing local attention operations across spatial and channel dimensions.

However, these computer vision models create an internal channel dimension without a semantic relation to an input dimension, whereas we design our input embedding and Transformer layers to preserve the relationship of the event dimension to input events. In dense time-series forecasting, TSMixer #OTHEREFR

(2023) has shown the promise of mixing across dimensions with MLP models.

Recently, Crossformer Zhang and Yan (2023) has introduced a modified attention function to mix across channels in a dense time-series forecasting Transformer model, but does not apply full quadratic attention or feedforward processing across the channel dimension as our model does.

SSL has become an important framework to enable learning useful representations from data without relying on labels.

Different approaches for SSL with numeric EHR data are explored in #OTHEREFR

(2021) , but these are applied to a relatively basic GRU model and do not claim to achieve state-of-the-art results.

mTAN uses a similar approach to SSL based on reconstructing inputs, incorporating a variational loss into their training procedure, but without a distinct pre-training stage.

SSL is used with Transformers in Hi-BEHRT, which applies BYOL #OTHEREFR to augmentations of EHR time series data.

STraTs uses masked value prediction for SSL with Transformers, while RAPT additionally uses a reasonability check to identify corrupted sequences and a contrastive patient similarity task.

Duett: Dual Event Time Transformer For Electronic Health Records

Abstract: Electronic health records (EHRs) recorded in hospital settings typically contain a wide range of numeric time series data that is characterized by high sparsity and irregular observations. Effective modelling for such data must exploit its time series nature, the semantic relationship between different types of observations, and information in the sparsity structure of the data. Self-supervised Transformers have shown outstanding performance in a variety of structured tasks in NLP and computer vision. But multivariate time series data contains structured relationships over two dimensions: time and recorded event type, and straightforward applications of Transformers to time series data do not leverage this distinct structure. The quadratic scaling of self-attention layers can also significantly limit the input sequence length without appropriate input engineering. We introduce the DuETT architecture, an extension of Transformers designed to attend over both time and event type dimensions, yielding robust representations from EHR data. DuETT uses an aggregated input where sparse time series are transformed into a regular sequence with fixed length; this lowers the computational complexity relative to previous EHR Transformer models and, more importantly, enables the use of larger and deeper neural networks. When trained with self-supervised prediction tasks, that provide rich and informative signals for model pre-training, our model outperforms state-of-the-art deep learning models on multiple downstream tasks from the MIMIC-IV and PhysioNet-2012 EHR datasets.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


Evaluation On Icfhr 2016 Competition'S Datasets

In summary, we note that the ST-KeyS semi-supervised model outperforms the fully supervised PHOC-based methods #REFR that were conducted in the literature on the BOT dataset and yields moderate results with these methods on the AK dataset.

From Table 3 , we notice that the proposed ST-KeyS model yields good results on the BOT dataset reaching a performance of 59.10% which outperforms TPP-PHOCNet(CPS), PHOCNet(CPS) and Triplet CNN architectures #OTHEREFR .

However, we obtained moderate results on the AK dataset comparing to TPP-PHOCNet(CPS) and PHOCNet(CPS) architectures #OTHEREFR .

Moreover, we observe that the ST-KeyS leads to better results than learningfree based methods #OTHEREFR on both the BOT and AK datasets although they represent different calligraphic styles compared to the IAM database utilized in the first pretraining stage and also despite the difference in the written language script.

Additionally, our ST-KeyS model gives better results compared to the Graph Edit Distance method proposed in #OTHEREFR thanks to the transformer's capacity to focus on meaningful information included in the word image.

This capacity is owed to the attention mechanism allowing to extract discriminant features from word images and which are used later in the matching process.

Generally, these methods rely on synthetically generated data to learn the PHOC contextual representations.

Thus, such models are able to treat large vocabulary allowing to alleviate to the out of vocabulary words issue #OTHEREFR .

In the remainder of our work, we limit the PHOC learning stage to the data provided in the competition image databases.

In other words, we are based on a closed vocabulary since our goal is to build a robust model allowing to retrieve handwritten words independently of the out of vocabulary word problem.

Therefore, it is unfair to compare our results to PHOC-based methods that were conducted in the literature.

St-Keys : Self-Supervised Transformer For Keyword Spotting In Historical Handwritten Documents

Abstract: Keyword spotting (KWS) in historical documents is an important tool for the initial exploration of digitized collections. Nowadays, the most efficient KWS methods are relying on machine learning techniques that require a large amount of annotated training data. However, in the case of historical manuscripts, there is a lack of annotated corpus for training. To handle the data scarcity issue, we investigate the merits of the self-supervised learning to extract useful representations of the input data without relying on human annotations and then using these representations in the downstream task. We propose ST-KeyS, a masked auto-encoder model based on vision transformers where the pretraining stage is based on the mask-and-predict paradigm, without the need of labeled data. In the fine-tuning stage, the pre-trained encoder is integrated into a siamese neural network model that is fine-tuned to improve feature embedding from the input images. We further improve the image representation using pyramidal histogram of characters (PHOC) embedding to create and exploit an intermediate representation of images based on text attributes. In an exhaustive experimental evaluation on three widely used benchmark datasets (Botany, Alvermann Konzilsprotokolle and George Washington), the proposed approach outperforms state-of-the-art methods trained on the same datasets.

Attribute Cnns For Word Spotting In Handwritten Documents

Abstract: Word spotting has become a field of strong research interest in document image analysis over the last years. Recently, AttributeSVMs were proposed which predict a binary attribute representation. At their time, this influential method defined the state-of-the-art in segmentation-based word spotting. In this work, we present an approach for learning attribute representations with Convolutional Neural Networks (CNNs). By taking a probabilistic perspective on training CNNs, we derive two different loss functions for binary and real-valued word string embeddings. In addition, we propose two different CNN architectures, specifically designed for word spotting. These architectures are able to be trained in an end-to-end fashion. In a number of experiments, we investigate the influence of different word string embeddings and optimization strategies. We show our Attribute CNNs to achieve state-of-the-art results for segmentation-based word spotting on a large variety of data sets.


It is worth noting that stability is of great importance for optimizing over the QAOA parameter landscape, as the great number of local minima requires generically exponential independent optimization trials to reach a global minimum #REFR .

2 shows the results for a collection of different optimizers, including Adam #OTHEREFR , COBYLA #OTHEREFR , and DARBO.

(For benchmark with more choices of common optimizers, see the Supplemental Material.) The results are collected from five different n = 16 w3R graph instances, and for each graph and each optimization method, the best record over 20 independent optimization trials is reported.

The results show the superior efficiency of DARBO over other common optimizers.

For all different p values, the approximation gaps 1 − r of Adam and COBYLA are 1.02 ∼ 2.08 times and 1.28 ∼ 3.47 times larger than those of DARBO, respectively.

In addition to the efficiency and accuracy, the DARBO performance is also more stable in terms of different problem graph instances.

As a result, with the increasing depth p of QAOA, the optimization problem becomes harder, and we see that Adam and COBYLA do not even exhibit a monotonic growth of the accuracy as p increases, despite the fact that the cut size given by the optimal |ψ(γ, β) (over the choice of 2p parameters (γ, β)) is clearly non-decreasing with p.

The reason of the performance drop is due to a lack of training stability and the sensitivity on initial parameter choices for conventional local optimization methods.

DARBO, on the contrary, does give better results with large depth p.

In other words, one important optimization advantage brought by DARBO is its capability of finding the near optimal parameters with a small number of independent optimization trials, which are not enough to locate the global optimal parameters for conventional optimizers.

Some advanced optimization methods such as FOURIER heuristics reported in #OTHEREFR can be good at locating global minimum but depend on progressive optimization on lower-depth QAOA, leading to a much larger total amount of required circuit evaluations.

Error-Mitigated Quantum Approximate Optimization Via Learning-Based Adaptive Optimization

Abstract: Combinatorial optimization problems are ubiquitous and computationally hard to solve in general. Quantum computing is envisioned as a powerful tool offering potential computational advantages for solving some of these problems. Quantum approximate optimization algorithm (QAOA), one of the most representative quantum-classical hybrid algorithms, is designed to solve certain combinatorial optimization problems by transforming a discrete optimization problem into a classical optimization problem over a continuous circuit parameter domain. QAOA objective landscape over the parameter variables is notorious for pervasive local minima and barren plateaus, and its viability in training significantly relies on the efficacy of the classical optimization algorithm. To enhance the performance of QAOA, we design double adaptive-region Bayesian optimization (DARBO), an adaptive classical optimizer for QAOA. Our experimental results demonstrate that the algorithm greatly outperforms conventional gradient-based and gradient-free optimizers in terms of speed, accuracy, and stability. We also address the issues of measurement efficiency and the suppression of quantum noise by successfully conducting the full optimization loop on the superconducting quantum processor. This work helps to unlock the full power of QAOA and paves the way toward achieving quantum advantage in practical classical tasks.

Quantum Approximate Optimization Algorithm: Performance, Mechanism, And Implementation On Near-Term Devices

Abstract: The Quantum Approximate Optimization Algorithm (QAOA) is a hybrid quantum-classical variational algorithm designed to tackle combinatorial optimization problems. Despite its promise for near-term quantum applications, not much is currently understood about QAOA's performance beyond its lowest-depth variant. An essential but missing ingredient for understanding and deploying QAOA is a constructive approach to carry out the outer-loop classical optimization. We provide an in-depth study of the performance of QAOA on MaxCut problems by developing an efficient parameter-optimization procedure and revealing its ability to exploit non-adiabatic operations. Building on observed patterns in optimal parameters, we propose heuristic strategies for initializing optimizations to find quasi-optimal $p$-level QAOA parameters in $O(\text{poly}(p))$ time, whereas the standard strategy of random initialization requires $2^{O(p)}$ optimization runs to achieve similar performance. We then benchmark QAOA and compare it with quantum annealing, especially on difficult instances where adiabatic quantum annealing fails due to small spectral gaps. The comparison reveals that QAOA can learn via optimization to utilize non-adiabatic mechanisms to circumvent the challenges associated with vanishing spectral gaps. Finally, we provide a realistic resource analysis on the experimental implementation of QAOA. When quantum fluctuations in measurements are accounted for, we illustrate that optimization will be important only for problem sizes beyond numerical simulations, but accessible on near-term devices. We propose a feasible implementation of large MaxCut problems with a few hundred vertices in a system of 2D neutral atoms, reaching the regime to challenge the best classical algorithms.


This is realized by a BERT #REFR encoder, a composition of layers for input embedding, positional encoding and multi-head self-attention blocks (cf.

Differently to the GAD characterization and aggregation function (cf. Eq. 3 and Eq.

4) of the deep generative models of #OTHEREFR our deep bidirectional GAD-Former Ψ models the characterization and aggregation functions as follows with

The characterization function f θ of GADFormer maps the bidirectional relationships between group members o i of a group G m (representations of trajectory point p or segment s of trajectory Λ m ) to an attention-based feature map b Gm , representing the behavior of an individual group (the individual trajectory path).

Figure 1 ) using group member embeddings pe as input tokens.

The aggregation function g Φ of GADFormer approximates instead of a distribution for normal group representations G (ref ) with distance measure d a probability p m for abnormal group behavior (abnormal trajectory path).

This is realized by non-linear custom task layers of the output block, which maps the group behavior characteristics b Λm to a task specific feature map representation z Λm .

Gadformer: An Attention-Based Model For Group Anomaly Detection On Trajectories

Abstract: Group Anomaly Detection (GAD) reveals anomalous behavior among groups consisting of multiple member instances, which are, individually considered, not necessarily anomalous. This task is of major importance across multiple disciplines, in which also sequences like trajectories can be considered as a group. However, with increasing amount and heterogenity of group members, actual abnormal groups get harder to detect, especially in an unsupervised or semi-supervised setting. Recurrent Neural Networks are well established deep sequence models, but recent works have shown that their performance can decrease with increasing sequence lengths. Hence, we introduce with this paper GADFormer, a GAD specific BERT architecture, capable to perform attention-based Group Anomaly Detection on trajectories in an unsupervised and semi-supervised setting. We show formally and experimentally how trajectory outlier detection can be realized as an attention-based Group Anomaly Detection problem. Furthermore, we introduce a Block Attention-anomaly Score (BAS) to improve the interpretability of transformer encoder blocks for GAD. In addition to that, synthetic trajectory generation allows us to optimize the training for domain-specific GAD. In extensive experiments we investigate our approach versus GRU in their robustness for trajectory noise and novelties on synthetic and real world datasets.


A naive solution would be to use recurrent neural networks (RNNs) such as Conv-LSTM #REFR to extract the dynamics from the input frames and predict the next, but this solution yields poor results.

Since we focus on segmentation tasks, we directly use the binary mask of an instance to represent the shape and position and predict motion upon it.

Let I t ∈ R w×h×3 denotes the t-th frame of the input video and m k t ∈ R w×h denotes the binary mask of instance k in this frame.

The goal of this module is to learn the motion and deformation from m k t−n to m k t−1 , and then predict the shape and position, denoted by m k t .

In our problem, we only require our module to predict one frame forward, but we wish it can take various number of frames as input, i.e.

m k t = F (m k t−n:t−1 ), ∀n ≥ 2 where F denotes our motion module.

They tend to make pixel-wise predictions for the entire image, while we want to learn instance-level motion and thus all pixels should be considered as a unit.

Following the concept of conditional variational autoencode (CVAE) that models the dynamics with a Gaussian and samples latent variables z t from this distribution for prediction #OTHEREFR , we also use an encoder during training to learn the motion pattern represented by z t .

However, we maintain a memory network M to store representative motion patterns.

The stored motion patterns are then used during inference to help refine the incomplete pattern that is extracted only from previous frames m k t−n:t−1 , and the refined pattern, which contains the instance-level motion information, is used to assist the prediction of the next frames m k t . Fig.

2 shows the overall framework of our motion prediction plugin.

Instmove: Instance Motion For Object-Centric Video Segmentation

Abstract: Despite significant efforts, cutting-edge video segmentation methods still remain sensitive to occlusion and rapid movement, due to their reliance on the appearance of objects in the form of object embeddings, which are vulnerable to these disturbances. A common solution is to use optical flow to provide motion information, but essentially it only considers pixel-level motion, which still relies on appearance similarity and hence is often inaccurate under occlusion and fast movement. In this work, we study the instance-level motion and present InstMove, which stands for Instance Motion for Object-centric Video Segmentation. In comparison to pixel-wise motion, Inst-Move mainly relies on instance-level motion information that is free from image feature embeddings, and features physical interpretations, making it more accurate and robust toward occlusion and fast-moving objects. To better fit in with the video segmentation tasks, InstMove uses instance masks to model the physical presence of an object and learns the dynamic model through a memory network to predict its position and shape in the next frame. With only a few lines of code, InstMove can be integrated into current SOTA methods for three different video segmentation tasks and boost their performance. Specifically, we improve the previous arts by 1.5 AP on OVIS dataset, which features heavy occlusions, and 4.9 AP on YouTubeVIS-Long dataset, which mainly contains fast moving objects. These results suggest that instance-level motion is robust and accurate, and hence serving as a powerful solution in complex scenarios for object-centric video segmentation.

Convolutional Lstm Network: A Machine Learning Approach For Precipitation Nowcasting

Abstract: The goal of precipitation nowcasting is to predict the future rainfall intensity in a local region over a relatively short period of time. Very few previous studies have examined this crucial and challenging weather forecasting problem from the machine learning perspective. In this paper, we formulate precipitation nowcasting as a spatiotemporal sequence forecasting problem in which both the input and the prediction target are spatiotemporal sequences. By extending the fully connected LSTM (FC-LSTM) to have convolutional structures in both the input-to-state and state-to-state transitions, we propose the convolutional LSTM (ConvLSTM) and use it to build an end-to-end trainable model for the precipitation nowcasting problem. Experiments show that our ConvLSTM network captures spatiotemporal correlations better and consistently outperforms FC-LSTM and the state-of-the-art operational ROVER algorithm for precipitation nowcasting.


The SDLGM is compared with state-of-the-art methods, including Gumbel-Softmax, ST Gumbel-Softmax, Score-Function (SF), DARN, MuProp, Straight-Through (ST), and Slope-Annealed ST #REFR .

We start with the representation learning task.

The performance of the methods is measured with negative variational lower bound of the log-likelihood (lower is better), which is a commonly used criterion in the literature.

In the experiments, we follow the similar settings as #OTHEREFR . The popular image dataset MNIST is used.

The encoder is defined as We first quantitatively demonstrate the performance of the proposed method with the commonly used measurement: the negative variational lower bound (lower is better).

We compare the SDLGM with multiple baselines, shown as Table I .

For a fair comparison, we directly use the results reported in #OTHEREFR . More details of the baselines refer to #OTHEREFR .

Learning Sparsity Of Representations With Discrete Latent Variables

Abstract: Deep latent generative models have attracted increasing attention due to the capacity of combining the strengths of deep learning and probabilistic models in an elegant way. The data representations learned with the models are often continuous and dense. However in many applications, sparse representations are expected, such as learning sparse high dimensional embedding of data in an unsupervised setting, and learning multi-labels from thousands of candidate tags in a supervised setting. In some scenarios, there could be further restriction on degree of sparsity: the number of non-zero features of a representation cannot be larger than a pre-defined threshold L0. In this paper we propose a sparse deep latent generative model SDLGM to explicitly model degree of sparsity and thus enable to learn the sparse structure of the data with the quantified sparsity constraint. The resulting sparsity of a representation is not fixed, but fits to the observation itself under the pre-defined restriction. In particular, we introduce to each observation i an auxiliary random variable Li, which models the sparsity of its representation. The sparse representations are then generated with a two-step sampling process via two Gumbel-Softmax distributions. For inference and learning, we develop an amortized variational method based on MC gradient estimator. The resulting sparse representations are differentiable with backpropagation. The experimental evaluation on multiple datasets for unsupervised and supervised learning problems shows the benefits of the proposed method.

Categorical Reparameterization With Gumbel-Softmax

Abstract: Categorical variables are a natural choice for representing discrete structure in the world. However, stochastic neural networks rarely use categorical latent variables due to the inability to backpropagate through samples. In this work, we present an efficient gradient estimator that replaces the non-differentiable sample from a categorical distribution with a differentiable sample from a novel Gumbel-Softmax distribution. This distribution has the essential property that it can be smoothly annealed into a categorical distribution. We show that our Gumbel-Softmax estimator outperforms state-of-the-art gradient estimators on structured output prediction and unsupervised generative modeling tasks with categorical latent variables, and enables large speedups on semi-supervised classification.


Interpolation Strategies We compare BCP with other two interpolation strategies: Mixup #REFR and Fine-Grained CutMix (FG-CutMix).

Inward and outward copy-paste (In and Out in the table) mean using

(1 − M) respectively to train the network.

We also conduct within-set copy-paste (CP in the table), i.e., copy-paste labeled data on another labeled data and copy-paste unlabeled data on another unlabeled data.

We can see that all these variants get inferior performances compared with our BCP, since they either lack consistent manner for training labeled and unlabeled data, or lack common semantics transfer between labeled and unlabeled data.

For Mixup, we superimpose labeled and unlabeled data to generate new training images, imitating the framework of GuidedMix-Net #OTHEREFR .

For FG-CutMix, we crop training images into 4×4 patches and combine them in batch to generate new images.

The network predictions of new images are re-combined and then supervised by ground-truth or pseudo-labels.

The results of LA dataset are shown in Table 5 .

Due to similar spatial structures of medical images, Mixup brings more influential noise in medical images. FG-CutMix maintains less structure information after CutMix than BCP. More details will be discussed in supplementary material. Fig.

Bidirectional Copy-Paste For Semi-Supervised Medical Image Segmentation

Abstract: In semi-supervised medical image segmentation, there exist empirical mismatch problems between labeled and unlabeled data distribution. The knowledge learned from the labeled data may be largely discarded if treating labeled and unlabeled data separately or in an inconsistent manner. We propose a straightforward method for alleviating the problem − copy-pasting labeled and unlabeled data bidirectionally, in a simple Mean Teacher architecture. The method encourages unlabeled data to learn comprehensive common semantics from the labeled data in both inward and outward directions. More importantly, the consistent learning procedure for labeled and unlabeled data can largely reduce the empirical distribution gap. In detail, we copypaste a random crop from a labeled image (foreground) onto an unlabeled image (background) and an unlabeled image (foreground) onto a labeled image (background), respectively. The two mixed images are fed into a Student network and supervised by the mixed supervisory signals of pseudo-labels and ground-truth. We reveal that the simple mechanism of copy-pasting bidirectionally between labeled and unlabeled data is good enough and the experiments show solid gains (e.g., over 21% Dice improvement on ACDC dataset with 5% labeled data) compared with other state-of-the-arts on various semi-supervised medical image segmentation datasets. Code is available at https: //github.com/DeepMed-Lab-ECNU/BCP.

Mixup: Beyond Empirical Risk Minimization

Abstract: Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.


Recent progressives in federated learning (FL) #REFR have proved that utilizing distributed data sources can both preserve data privacy and enhance model training.

#OTHEREFR How to improve the question answering accuracy has become a serious challenge which decides whether LLMs can be accepted as a robust and reliable part in realistic applications.

One commonsense is that can we crowd-source many questions and aggregate those questions to better understand some common questions.

A common question might be asked frequently as its variants in the concrete parameters or rephrased formulations.

For example, the Chickens-and-rabbits questions can be asked with different number of heads and feet.

Now we want to ask Can we fully utilize those similar questions to improve the question answering of the LLMs without tuning the model parameters or infringing user privacy?

In the FL paradigm, each client trains a local learning model with own data, while a central server regularly communicates with all agents to generate a better global model through the aggregation of the local models.

In this study, we consider improving the reasoning capacity of LLMs by better understanding crowd-sourced similar questions, from which we can explore the contextual information and improve the LLM answers substantially.

Inspired by FL, we propose two typical scenarios when a user sends a QA request to the LLM and the LLM tries to answer with a collected question database.

-Synonymous Questions with Same Parameters (SP-questions).

The cloud-deployed system retrieves from the database and finds several synonymous but rephrased questions with exactly the same parameters.

Federated Prompting And Chain-Of-Thought Reasoning For Improving Llms Answering

Abstract: We investigate how to enhance answer precision in frequently asked questions posed by distributed users using cloud-based Large Language Models (LLMs). Our study focuses on a typical situations where users ask similar queries that involve identical mathematical reasoning steps and problem-solving procedures. Due to the unsatisfactory accuracy of LLMs' zero-shot prompting with standalone questions, we propose to improve the distributed synonymous questions using Self-Consistency (SC) and Chain-of-Thought (CoT) techniques. Specifically, we first retrieve synonymous questions from a crowd-sourced database and create a federated question pool. We call these federated synonymous questions with the same or different parameters SP-questions or DP-questions, respectively. We refer to our methods as Fed-SP-SC and Fed-DP-CoT, which can generate significantly more accurate answers for all user queries without requiring sophisticated model-tuning. Through extensive experiments, we demonstrate that our proposed methods can significantly enhance question accuracy by fully exploring the synonymous nature of the questions and the consistency of the answers.

Communication-Efficient Learning Of Deep Networks From Decentralized Data

Abstract: Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning. We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.

distributed data sources
federated learning


Furthermore, Dynamic bootstrapping #REFR use a Beta-Mixture-Model to combine label dynamically for every sample.

Therefore, various methods were proposed to improve robustness against noisy labels.

The typical algorithms for combating noisy labels including adding regularization #OTHEREFR , designing robust loss functions #OTHEREFR , selecting possible clean samples #OTHEREFR and correcting the labels #OTHEREFR .

We mainly introduce the label correction approaches which are most related to this work.

Specifically, label correction algorithms aim to identify and correct suspicious labels in an iterative framework.

Bootstrapping #OTHEREFR combines the noisy label and the DNN prediction as a soft pseudo-label to replace the original label.

Differently, Joint Optimization #OTHEREFR trained their model with a large learning rate for several epochs firstly, and then generated pseudo-labels by averaging the predictions of that epochs.

Recently, SELFIE #OTHEREFR and Ada-Corr #OTHEREFR selectively combine the labels of noisy data for corresponding to true labels with a high probability.

However, almost all existing noisy label studies, focusing on the classification task, cannot be adapted to cross-modal matching directly in which the noisy label refers to the alignment errors between data pairs rather than the categorical labeling errors.

The label of cross-modal matching can be consider as y i ∈ {0, 1}, where data pair is positively correlated (y i = 1) or not (y i = 0).

Therefore, label correction of noisy multi-modal data pairs can be translated into generating a continuous value between 0 and 1, called soft correspondence labels in this paper, to measure the true degree of correspondence.

Bicro: Noisy Correspondence Rectification For Multi-Modality Data Via Bi-Directional Cross-Modal Similarity Consistency

Abstract: As one of the most fundamental techniques in multimodal learning, cross-modal matching aims to project various sensory modalities into a shared feature space. To achieve this, massive and correctly aligned data pairs are required for model training. However, unlike unimodal datasets, multimodal datasets are extremely harder to collect and annotate precisely. As an alternative, the co-occurred data pairs (e.g., image-text pairs) collected from the Internet have been widely exploited in the area. Unfortunately, the cheaply collected dataset unavoidably contains many mismatched data pairs, which have been proven to be harmful to the model's performance. To address this, we propose a general framework called BiCro (Bidirectional Cross-modal similarity consistency), which can be easily integrated into existing cross-modal matching models and improve their robustness against noisy data. Specifically, BiCro aims to estimate soft labels for noisy data pairs to reflect their true correspondence degree. The basic idea of BiCro is motivated by that-taking image-text matching as an example-similar images should have similar textual descriptions and vice versa. Then the consistency of these two similarities can be recast as the estimated soft labels to train the matching model. The experiments on three popular cross-modal matching datasets demonstrate that our method significantly improves the noise-robustness of various matching models, and surpass the state-of-the-art by a clear margin.The code is available at https://github.com/xu5zhao/BiCro.

Unsupervised Label Noise Modeling And Loss Correction

Abstract: Despite being robust to small amounts of label noise, convolutional neural networks trained with stochastic gradient methods have been shown to easily fit random labels. When there are a mixture of correct and mislabelled targets, networks ::: tend to fit the former before the latter. This suggests using a suitable two-component mixture model as an unsupervised generative model of sample loss values during training to allow online estimation of the probability that a sample is mislabelled. Specifically, we propose a beta mixture to estimate this probability and correct the loss by relying on the network prediction (the so-called bootstrapping loss). We further adapt mixup augmentation to drive our approach a step further. Experiments on CIFAR-10/100 and TinyImageNet demonstrate a robustness to label noise that substantially outperforms recent state-of-the-art. Source code is available at https://git.io/fjsvE and Appendix at https://arxiv.org/abs/1904.11238.


On the other hand, participants in the Prediction condition failed to surpass the model's performance, suggesting that predictions alone are not as effective in improving the joint performance, supporting the findings in #REFR .

Figure 2a , depicts participants' unassisted performance per condition, along with a 95% confidence interval. Details about all CIs are presented in Appendix A.

Figure 2a shows that 75% belongs to all CIs, so participants and model showed comparable performance in all conditions, thus recreating the setting in #OTHEREFR .

Then, we compared participants' performance after exposure to the model's assistance (Assisted Performance) to the model's accuracy.

Figure 2a , shows the assisted performance, along with the corresponding 95% CIs.

Participants' assisted performance was significantly better than 75% in all but the Prediction condition, suggesting that even the simple strategy of pairing model predictions with confidence, as in the Local condition, is beneficial to participants' performance, in line with the findings in #OTHEREFR .

Having established that the model's assistance helped the ensemble surpass the individual model accuracy, we continue by examining whether it surpassed participant's individual accuracy as well.

Figure 2b , shows the 95% CIs of the difference between participants' assisted and unassisted performance, per condition.

Participants' assisted performance was significantly better than their unassisted performance in the Prediction and Local conditions.

On the contrary, the same comparison did not yield statistically significant results in the Combined and Explanations conditions, even though the point estimates were positive.

This pattern can be explained, at least in part, by the fact that participants in the Combined and Explanations conditions already had better performance in their unassisted predictions compared to participants in the Prediction and Local conditions, leaving less room for improvement for them.

Why Not Both? Complementing Explanations With Uncertainty, And The Role Of Self-Confidence In Human-Ai Collaboration

Abstract: AI and ML models have already found many applications in critical domains, such as healthcare and criminal justice. However, fully automating such high-stakes applications can raise ethical or fairness concerns. Instead, in such cases, humans should be assisted by automated systems so that the two parties reach a joint decision, stemming out of their interaction. In this work we conduct an empirical study to identify how uncertainty estimates and model explanations affect users' reliance, understanding, and trust towards a model, looking for potential benefits of bringing the two together. Moreover, we seek to assess how users' behaviour is affected by their own self-confidence in their abilities to perform a certain task, while we also discuss how the latter may distort the outcome of an analysis based on agreement and switching percentages. Preprint. Under review.

On Human Predictions With Explanations And Predictions Of Machine Learning Models: A Case Study On Deception Detection

Abstract: Humans are the final decision makers in critical tasks that involve ethical and legal concerns, ranging from recidivism prediction, to medical diagnosis, to fighting against fake news. Although machine learning models can sometimes achieve impressive performance in these tasks, these tasks are not amenable to full automation. To realize the potential of machine learning for improving human decisions, it is important to understand how assistance from machine learning models affects human performance and human agency. In this paper, we use deception detection as a testbed and investigate how we can harness explanations and predictions of machine learning models to improve human performance while retaining human agency. We propose a spectrum between full human agency and full automation, and develop varying levels of machine assistance along the spectrum that gradually increase the influence of machine predictions. We find that without showing predicted labels, explanations alone slightly improve human performance in the end task. In comparison, human performance is greatly improved by showing predicted labels (>20% relative improvement) and can be further improved by explicitly suggesting strong machine performance. Interestingly, when predicted labels are shown, explanations of machine predictions induce a similar level of accuracy as an explicit statement of strong machine performance. Our results demonstrate a tradeoff between human performance and human agency and show that explanations of machine predictions can moderate this tradeoff.


LLMs can also produce toxic or hateful content and fail to follow provided user-constraints #REFR .

We presented the OpenAssistant Conversations dataset, an outcome of a crowd-sourcing initiative aimed at promoting research in the area of alignment in LLMs.

We recognize that sufficiently powerful language models can have a significant impact on society #OTHEREFR , and therefore we believe it is essential to promote transparency and ethical considerations in their development and deployment.

These models are often prone to generating inaccurate information about people, places, or facts, a phenomenon commonly known as 'hallucinations' #OTHEREFR .

Additionally, these models tend to incorporate biases present in their training data, leading to unfair and discriminatory outputs #OTHEREFR .

While methods such as RLHF can mitigate some of these shortcomings, they may exacerbate others [26, 20] .

We hope that alignment can fix some of these issues #OTHEREFR , but it is important to acknowledge that achieving alignment is a complex and ongoing challenge.

Our team has put in significant effort to ensure that the community has access to an open-source high-quality dataset free of unethical or harmful responses.

We believe that creating a safe and respectful environment for our users is paramount, and we encourage them to generate prompts and replies that are not only polite, but also creative and detailed.

Openassistant Conversations -Democratizing Large Language Model Alignment

Abstract: Aligning large language models (LLMs) with human preferences has proven to drastically improve usability and has driven rapid adoption as demonstrated by ChatGPT. Alignment techniques such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) greatly reduce the required skill and domain knowledge to effectively harness the capabilities of LLMs, increasing their accessibility and utility across various domains. However, state-of-the-art alignment techniques like RLHF rely on high-quality human feedback data, which is expensive to create and often remains proprietary. In an effort to democratize research on large-scale alignment, we release OpenAssistant Conversations, a human-generated, human-annotated assistant-style conversation corpus consisting of 161,443 messages distributed across 66,497 conversation trees, in 35 different languages, annotated with 461,292 quality ratings. The corpus is a product of a worldwide crowd-sourcing effort involving over 13,500 volunteers. 2 To demonstrate the OpenAssistant Conversations dataset's effectiveness, we present OpenAssistant, the first fully open-source large-scale instruction-tuned model to be trained on human data. A preference study revealed that OpenAssistant replies are comparably preferred to GPT-3.5-turbo (ChatGPT) with a relative winrate of 48.3% vs. 51.7% respectively. We release our code 3 and data 4 under fully permissive licenses.

Ethical Challenges In Data-Driven Dialogue Systems

Abstract: The use of dialogue systems as a medium for human-machine interaction is an increasingly prevalent paradigm. A growing number of dialogue systems use conversation strategies that are learned from large datasets. There are well documented instances where interactions with these system have resulted in biased or even offensive conversations due to the data-driven training process. Here, we highlight potential ethical issues that arise in dialogue systems research, including: implicit biases in data-driven systems, the rise of adversarial examples, potential sources of privacy violations, safety concerns, special considerations for reinforcement learning systems, and reproducibility concerns. We also suggest areas stemming from these issues that deserve further investigation. Through this initial survey, we hope to spur research leading to robust, safe, and ethically sound dialogue systems.


Only (ImageNet) #REFR 5 d on the model performance, we conduct an additional ablation study of d on the iNat2018 dataset with different λ and report the results in Table 3 .

To understand the effectiveness of each loss component in CSP (see Figure 3 and Equation 4), we conduct an ablation study on the iNat2018 dataset with different λ and report the results in Table 2 .

We can see that each component contributes to the final model performance. Deleting any of them will lead to performance drops.

To understand the effect of location embedding dimension 4 https://github.com/visipedia/inat_comp/ tree/master/2018 Table 1 : The Top1 accuracy of different models and training strategies on the iNat2018 validation dataset for the species finegrain recognition task with different training data ratios, where λ% = 100% indicates the fully supervised setting.

We run each model 5 times and report the standard deviation in "()".

We can see that at the few-shot setting λ% = 5%, 10%, 20%, models with d = 256 achieve the best performance.

In the fully supervised setting, the model with d = 1024 leads to the best performance.

Last but not least, we also explore whether our CSP is effective on different image encoders.

We conduct an ablation study of different F() on the iNat2018 dataset with λ% = 5%. Table 4 summarizes the results.

We can see that no matter which F() we use, Inception V3 or ViT, our CSP-MC-BLD consistently yields the best results, and ViT improves the model performance a lot.

Csp: Self-Supervised Contrastive Spatial Pre-Training For Geospatial-Visual Representations

Abstract: Geo-tagged images are publicly available in large quantities, whereas labels such as object classes are rather scarce and expensive to collect. Meanwhile, contrastive learning has achieved tremendous success in various natural image and language tasks with limited labeled data. However, existing methods fail to fully leverage geospatial information, which can be paramount to distinguishing objects that are visually similar. To directly leverage the abundant geospatial information associated with images in pre-training, fine-tuning, and inference stages, we present Contrastive Spatial Pre-Training (CSP), a selfsupervised learning framework for geo-tagged images. We use a dual-encoder to separately encode the images and their corresponding geo-locations, and use contrastive objectives to learn effective location representations from images, which can be transferred to downstream supervised tasks such as image classification. Experiments show that CSP can improve model performance on both iNat2018 and fMoW dataset. Especially, on iNat2018, CSP significantly boosts the model performance with 10-34% relative improvement with various labeled training data sampling ratios 1 .

Rethinking The Inception Architecture For Computer Vision

Abstract: Convolutional networks are at the core of most state of-the-art computer vision solutions for a wide variety of tasks. Since 2014 very deep convolutional networks started to become mainstream, yielding substantial gains in various benchmarks. Although increased model size and computational cost tend to translate to immediate quality gains for most tasks (as long as enough labeled data is provided for training), computational efficiency and low parameter count are still enabling factors for various use cases such as mobile vision and big-data scenarios. Here we are exploring ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization. We benchmark our methods on the ILSVRC 2012 classification challenge validation set demonstrate substantial gains over the state of the art: 21:2% top-1 and 5:6% top-5 error for single frame evaluation using a network with a computational cost of 5 billion multiply-adds per inference and with using less than 25 million parameters. With an ensemble of 4 models and multi-crop evaluation, we report 3:5% top-5 error and 17:3% top-1 error on the validation set and 3:6% top-5 error on the official test set.


IITB-Corridor #REFR has one of the most extensive collections of frames but focuses on a single location which may not be generalized to other settings.

However, contemporary videobased methods #OTHEREFR discussed below were found to be more accurate, utilizing deep learning techniques.

Numerous methods have been proposed for detecting anomalous behaviors using video-based anomaly datasets.

These datasets are videos of normal and abnormal behaviors in the training and validation set.

One of the most recent datasets, Charlotte Anomaly Dataset (CHAD) [25] , provides high-resolution multi-camera anomaly detection with bounding box, identity, and pose annotations.

ShanghaiTech Campus dataset #OTHEREFR is a well-known dataset that provides a collection of 317K normal and anomalous frames across several scenes.

In contrast with previously mentioned datasets which collect videos from continuous streaming cameras, UCF Crime #OTHEREFR is based on the videos collected from YouTube, covering many scenes and thus changing the context with each scene.

This may not be ideal for anomaly detection as it is context-specific.

Another new dataset is the NOLA dataset #OTHEREFR , which consists of a single scene covered by a moving camera.

One of the characteristics of NOLA is that they use a moving camera, making them unique compared to their peers.

Unsupervised learning approaches for vision-based anomaly detection are well-established and widely used.

Understanding The Challenges And Opportunities Of Pose-Based Anomaly Detection

Abstract: Pose-based anomaly detection is a video-analysis technique for detecting anomalous events or behaviors by examining human pose extracted from the video frames. Utilizing pose data alleviates privacy and ethical issues. Also, computation-wise, the complexity of pose-based models is lower than pixel-based approaches. However, it introduces more challenges, such as noisy skeleton data, losing important pixel information, and not having enriched enough features. These problems are exacerbated by a lack of anomaly detection datasets that are good enough representatives of real-world scenarios. In this work, we analyze and quantify the characteristics of two well-known video anomaly datasets to better understand the difficulties of pose-based anomaly detection. We take a step forward, exploring the discriminating power of pose and trajectory for video anomaly detection and their effectiveness based on context. We believe these experiments are beneficial for a better comprehension of pose-based anomaly detection and the datasets currently available. This will aid researchers in tackling the task of anomaly detection with a more lucid perspective, accelerating the development of robust models with better performance.

Multi-Timescale Trajectory Prediction For Abnormal Human Activity Detection

Abstract: A classical approach to abnormal activity detection is to learn a representation for normal activities from the training data and then use this learned representation to detect abnormal activities while testing. Typically, the methods based on this approach operate at a fixed timescale - either a single time-instant (eg. frame-based) or a constant time duration (eg. video-clip based). But human abnormal activities can take place at different timescales. For example, jumping is a short term anomaly and loitering is a long term anomaly in a surveillance scenario. A single and pre-defined timescale is not enough to capture the wide range of anomalies occurring with different time duration. In this paper, we propose a multi-timescale model to capture the temporal dynamics at different timescales. In particular, the proposed model makes future and past predictions at different timescales for a given input pose trajectory. The model is multi-layered where intermediate layers are responsible to generate predictions corresponding to different timescales. These predictions are combined to detect abnormal activities. In addition, we also introduce an abnormal activity data-set for research use that contains 4,83,566 annotated frames. Data-set will be made available at this https URL Our experiments show that the proposed model can capture the anomalies of different time duration and outperforms existing methods.


The use of these proxy losses does not have a strong correlation with IoU #REFR . As noted by Yu et al.

Thus, the proxy loss function, in this case, is harsher on the model than the evaluation metric.

The mismatch between the per-character cross-entropy loss and the edit distance is shown in Figure 1 .1 through an example.

Given a natural scene image, the goal is to pre-cisely localize all instances of text at a word level. The ground truth consists of rotated bounding boxes.

Popular approaches use smooth-L 1 or L 2 distance for regressing bounding box coordinates.

#OTHEREFR , IoU accounts for a bounding box as a whole, whereas regressing using an L n proxy loss treats each point independently.

The objective of compression approaches is to minimize the storage cost of images.

In a lossy setting, this is achieved by removing information that is least noticeable to humans.

Learning-based compression approaches follow a rate-distortion objective, which tries to minimize the storage requirements while keeping the distortion as low as possible.

Abstract: I hereby declare I have written this doctoral dissertation independently and quoted all the sources of information used in accordance with methodological instructions on ethical principles for writing an academic dissertation. Moreover, I state that this dissertation has neither been submitted nor accepted for any other degree. The results presented in this dissertation were were achieved during my own research in cooperation with my dissertation supervisor Jiri Matas published in [1]-[8]. During the Ph.D., I also collaborated with several researchers on multiple projects, with R. Manmatha, Srikar

Generalized Intersection Over Union: A Metric And A Loss For Bounding Box Regression

Abstract: Intersection over Union (IoU) is the most popular evaluation metric used in the object detection benchmarks. However, there is a gap between optimizing the commonly used distance losses for regressing the parameters of a bounding box and maximizing this metric value. The optimal objective for a metric is the metric itself. In the case of axis-aligned 2D bounding boxes, it can be shown that IoU can be directly used as a regression loss. However, IoU has a plateau making it infeasible to optimize in the case of non-overlapping bounding boxes. In this paper, we address the this weakness by introducing a generalized version of IoU as both a new loss and a new metric. By incorporating this generalized IoU ( GIoU) as a loss into the state-of-the art object detection frameworks, we show a consistent improvement on their performance using both the standard, IoU based, and new, GIoU based, performance measures on popular object detection benchmarks such as PASCAL VOC and MS COCO.


In some specific problems, removing subgroup indicators such as the sex of a job candidate from their application has shown to have minimal influence on classifier accuracy while improving the fairness #REFR .

#OTHEREFR provide a detailed overview of data anonymization models and techniques in healthcare such as kanonymity, k-map, l-diversity, t-closeness, δ-disclosure privacy, β-likeness, δ-presence, and ( , δ)-differential privacy.

To avoid data leakage, many tools for data anonymization and its evaluation tools #OTHEREFR such as Sec-Graph #OTHEREFR , ARX-tool for anonymizing biomedical data #OTHEREFR , Amnesia 29 #OTHEREFR , PySyft #OTHEREFR , Synthea #OTHEREFR and Anonimatron 30 (open-source data anonymization tool written in Java) can be incorporated in the MLHOps pipeline.

Changing the race of the patients can have a dramatic impact on the outcome of an algorithm that is designed to fill a prompt #OTHEREFR .

Therefore, the existence of race attributes in the text can decrease the fairness of the model dramatically.

This method is applicable mostly in text-based data where sensitive attributes are easily removable.

As a preprocessing step, one can estimate the effect of keeping or removing such sensitive attributes on the overall accuracy and fairness of a developed model.

At the same time, it is not always possible to remove the sensitive attributes from the data.

For example, AI models can predict patient race from medical images, but it is not yet clear how they can do it #OTHEREFR .

In one study #OTHEREFR , researchers did not provide the patient race during model training, but they also could not find a particular patch or region in the data for which AI failed to detect race by removing that part.

Mlhops: Machine Learning For Healthcare Operations

Abstract: Machine Learning Health Operations (MLHOps) is the combination of processes for reliable, efficient, usable, and ethical deployment and maintenance of machine learning models in healthcare settings. This paper provides both a survey of work in this area and guidelines for developers and clinicians to deploy and maintain their own models in clinical practice. We cover the foundational concepts of general machine learning operations, describe the initial setup of MLHOps pipelines (including data sources, preparation, engineering, and tools). We then describe long-term monitoring and updating (including data distribution shifts and model updating) and ethical considerations (including bias, fairness, interpretability, and privacy). This work therefore provides guidance across the full pipeline of MLHOps from conception to initial and ongoing deployment.

Bias In Bios: A Case Study Of Semantic Representation Bias In A High-Stakes Setting

Abstract: We present a large-scale study of gender bias in occupation classification, a task where the use of machine learning may lead to negative outcomes on peoples' lives. We analyze the potential allocation harms that can result from semantic representation bias. To do so, we study the impact on occupation classification of including explicit gender indicators---such as first names and pronouns---in different semantic representations of online biographies. Additionally, we quantify the bias that remains when these indicators are "scrubbed," and describe proxy behavior that occurs in the absence of explicit gender indicators. As we demonstrate, differences in true positive rates between genders are correlated with existing gender imbalances in occupations, which may compound these imbalances.


A similar behavior is observed for Mix+Incons_hardMag, which outperforms its special case σ = 0 #REFR .

First, let us investigate the influence of the consistency wiehgt σ onto performance. From the results displayed in Fig.

1a-1c , we remark that Mag+Incons_hardMix exhibits a stable performance with respect to σ.

Conversely, for the other algorithms, the SDR peaks at specific values of σ.

In particular, a properly tuned Mix+Incons outperforms its special cases corresponding to σ = 0 and +∞, which were use in #OTHEREFR .

This demonstrates the interest of our framework, which allows for deriving general algorithms that outperform their special cases found in the literature. Besides, Fig.

1d reveals that the algorithms exhibit a different behavior over iterations.

Indeed, while for most methods, the SDR steadily increases over iterations, MISI reaches its peak performance after very few iterations, and then its performance drops severely.

This was notably observed in #OTHEREFR ; nonetheless, the modified MISI algorithm introduced in this paper (which is similar to Mix+Incons_hardMag) was compared to MISI after 200 iterations, which is somewhat unfair to MISI.

Here, we select the optimal weight and number of iterations for each algorithm in order to run them on the test set in a more fair setup.

Spectrogram Inversion For Audio Source Separation Via Consistency, Mixing, And Magnitude Constraints

Abstract: Audio source separation is often achieved by estimating the magnitude spectrogram of each source, and then applying a phase recovery (or spectrogram inversion) algorithm to retrieve time-domain signals. Typically, spectrogram inversion is treated as an optimization problem involving one or several terms in order to promote estimates that comply with a consistency property, a mixing constraint, and/or a target magnitude objective. Nonetheless, it is still unclear which set of constraints and problem formulation is the most appropriate in practice. In this paper, we design a general framework for deriving spectrogram inversion algorithm, which is based on formulating optimization problems by combining these objectives either as soft penalties or hard constraints. We solve these by means of algorithms that perform alternating projections on the subsets corresponding to each objective/constraint. Our framework encompasses existing techniques from the literature as well as novel algorithms. We investigate the potential of these approaches for a speech enhancement task. In particular, one of our novel algorithms outperforms other approaches in a realistic setting where the magnitudes are estimated beforehand using a neural network.

Model-Based Stft Phase Recovery For Audio Source Separation

Abstract: For audio source separation applications, it is common to estimate the magnitude of the short-time Fourier transform (STFT) of each source. In order to further synthesizing time-domain signals, it is necessary to recover the phase of the corresponding complex-valued STFT. Most authors in this field choose a Wiener-like filtering approach which boils down to using the phase of the original mixture. In this paper, a different standpoint is adopted. Many music events are partially composed of slowly varying sinusoids and the STFT phase increment over time of those frequency components takes a specific form. This allows phase recovery by an unwrapping technique once a short-term frequency estimate has been obtained. Herein, a novel iterative source separation procedure is proposed which builds upon these results. It consists in minimizing the mixing error by means of the auxiliary function method. This procedure is initialized by exploiting the unwrapping technique in order to generate estimates that benefit from a temporal continuity property. Experiments conducted on realistic music pieces show that, given accurate magnitude estimates, this procedure outperforms the state-of-the-art consistent Wiener filter.


We explore the synthetic and KDDCUP'99 datasets since these are what were tested in the private ADMM paper #REFR .

All reported ǫ values indicate the privacy of the output weights, ǫ 1 + ǫ 2 . Since we were unable to Fig. 1 .

Comparing the number of correctly identified zeros of Sparsifier with twostage, private ADMM, and private IGHT. Sparsifier and private IGHT were run for 1000 iterations.

A black line at 92 coefficients has been drawn indicating the number of true zero coefficients.

Higher numbers of correct zero coefficients indicate that an algorithm is able to maintain sparsity.

replicate the sparse outputs of the private ADMM algorithm, we directly use the results provided in that paper.

Synthetic Data Here, we test Sparsifier on a synthetic dataset to demonstrate that it outperforms prior solutions for sparse private logistic regression.

We compare our solution to the the two-stage procedure developed by Kifer et al., private ADMM algorithm developed by Wang & Zhang, and the private IGHT method developed by Wang & Gu #OTHEREFR .

For this comparison, we generated synthetic data according to the details provided by Wang & Zhang #OTHEREFR .

Specifically, we generated 10, 000 datapoints where each x i ∈ R 100 was drawn from N (0, Σ) with Σ ij = 0.5 |i−j| and scaled each feature so its maximum absolute value in the dataset was 1.

The true weight vector w * had its first 8 components set to 10 9 8 7 6 5 4 0.5 and its last 92 components set to 0.

Sparse Private Lasso Logistic Regression

Abstract: LASSO regularized logistic regression is particularly useful for its built-in feature selection, allowing coefficients to be removed from deployment and producing sparse solutions. Differentially private versions of LASSO logistic regression have been developed, but generally produce dense solutions, reducing the intrinsic utility of the LASSO penalty. In this paper, we present a differentially private method for sparse logistic regression that maintains hard zeros. Our key insight is to first train a non-private LASSO logistic regression model to determine an appropriate privatized number of non-zero coefficients to use in final model selection. To demonstrate our method's performance, we run experiments on synthetic and real-world datasets.

Differential Privacy For Sparse Classification Learning

Abstract: In this paper, we present a differential privacy version of convex and nonconvex sparse classification approach. Based on alternating direction method of multiplier (ADMM) algorithm, we transform the solving of sparse problem into the multistep iteration process. Then we add exponential noise to stable steps to achieve privacy protection. By the property of the post-processing holding of differential privacy, the proposed approach satisfies the ϵ − differential privacy even when the original problem is unstable. Furthermore, we present the theoretical privacy bound of the differential privacy classification algorithm. Specifically, the privacy bound of our algorithm is controlled by the algorithm iteration number, the privacy parameter, the parameter of loss function, ADMM pre-selected parameter, and the data size. Finally we apply our framework to logistic regression with L1 regularizer and logistic regression with L1/2 regularizer. Numerical studies demonstrate that our method is both effective and efficient which performs well in sensitive data analysis.


Since AffineLayer performs significantly worse than BatchNorm, these results cast doubt on the claim that the success of BatchNorm is due to it stabilizing the gradients #REFR .

Here, we extract the weights and gradients for each of input-early, input-late, final-early, and final-late, and we plot various histograms of these distributions.

In Figure 3(a) , we see that the weight distribution changes from input-early to final-late are fairly modest for BatchNorm, while Figure 3 (b) yields the same conclusion for AffineLayer.

In Figure 3 (c), we have overlayed the final-late weights of BatchNorm and AffineLayer, while Figure 3(d) gives the analogous result for the gradients.

We see that the weights and gradients behave similarly, and hence whatever is causing BatchNorm to outperform AffineLayer does not appear to be distinguishable through this weight and gradient distribution comparison.

The bottom line here is that AffineLayer has a similar effect on the gradients as BatchNorm, at least in the all-important first epoch.

The results in Figure 3 show minimal differences between BatchNorm and AffineLayer.

A significant difference between AffineLayer and BatchNorm can be observed by comparing the input-early and input-late updates.

Specifically, in Figure 4(a) , we compare the input-early gradient with the input-late gradient for BatchNorm, while Figure 4 (b) provides the analogous comparison for AffineLayer.

We observe from Figure 4 (a) that for BatchNorm, the gradient starts spread out and become more tightly focused, whereas Figure 4(b) shows the opposite behavior for the AffineLayer gradient.

Figure 4 provides additional evidence that it is the weight normalization at the beginning of each epoch that is causing the observed performance differences between BatchNorm and AffineLayer, at least in the case of ResNet18. We discuss this further in the next section.

An Empirical Analysis Of The Shift And Scale Parameters In Batchnorm

Abstract: Batch Normalization (BatchNorm) is a technique that improves the training of deep neural networks, especially Convolutional Neural Networks (CNN). It has been empirically demonstrated that BatchNorm increases performance, stability, and accuracy, although the reasons for such improvements are unclear. BatchNorm includes a normalization step as well as trainable shift and scale parameters. In this paper, we empirically examine the relative contribution to the success of BatchNorm of the normalization step, as compared to the re-parameterization via shifting and scaling. To conduct our experiments, we implement two new optimizers in PyTorch, namely, a version of BatchNorm that we refer to as AffineLayer, which includes the re-parameterization step without normalization, and a version with just the normalization step, that we call BatchNorm-minus. We compare the performance of our AffineLayer and BatchNorm-minus implementations to standard BatchNorm, and we also compare these to the case where no batch normalization is used. We experiment with four ResNet architectures (ResNet18, ResNet34, ResNet50, and ResNet101) over a standard image dataset and multiple batch sizes. Among other findings, we provide empirical evidence that the success of BatchNorm may derive primarily from improved weight initialization.

Batch Normalization: Accelerating Deep Network Training By Reducing Internal Covariate Shift

Abstract: Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.


To verify our proposed novel RCCV is a general 3D geometric feature representation and not limited to a specific framework, we additionally apply it to Atlas #REFR to replace its simple back-projected feature. The results meet our expectations.

The lower the better The higher the better Accuracy Completeness Chamfer Precision Recall F1 Score Atlas #OTHEREFR Table 4 . Atlas with Our RCCV.

3 sets of experiments are designed as follows: #OTHEREFR We simply concatenate the keyframe 2D image feature to the Ray-compensated Cost Volume.

All 3D mesh metrics are improved by this concatenation, confirming the effectiveness of the contextual feature.

(2)A 2D convolutional layer is employed to better fuse the contextual feature with the cost volume, the improvements are relatively minor.

3We use a group convolution layer to limit the perceptive field of the depth planes in the cost volume to be within itself.

Cvrecon: Rethinking 3D Geometric Feature Learning For Neural Reconstruction

Abstract: Recent advances in neural reconstruction using posed image sequences have made remarkable progress. However, due to the lack of depth information, existing volumetric-based techniques simply duplicate 2D image features of the object surface along the entire camera ray. We contend this duplication introduces noise in empty and occluded spaces, posing challenges for producing highquality 3D geometry. Drawing inspiration from traditional multi-view stereo methods, we propose an end-to-end 3D neural reconstruction framework CVRecon, designed to exploit the rich geometric embedding in the cost volumes to facilitate 3D geometric feature learning. Furthermore, we present Ray-contextual Compensated Cost Volume (RCCV), a novel 3D geometric feature representation that encodes view-dependent information with improved integrity and robustness. Through comprehensive experiments, we demonstrate that our approach significantly improves the reconstruction quality in various metrics and recovers clear fine details of the 3D geometries. Our extensive ablation studies provide insights into the development of effective 3D geometric feature learning schemes. Project page: https://cvrecon.ziyue.cool/.

Atlas: End-To-End 3D Scene Reconstruction From Posed Images

Abstract: We present an end-to-end 3D reconstruction method for a scene by directly regressing a truncated signed distance function (TSDF) from a set of posed RGB images. Traditional approaches to 3D reconstruction rely on an intermediate representation of depth maps prior to estimating a full 3D model of a scene. We hypothesize that a direct regression to 3D is more effective. A 2D CNN extracts features from each image independently which are then back-projected and accumulated into a voxel volume using the camera intrinsics and extrinsics. After accumulation, a 3D CNN refines the accumulated features and predicts the TSDF values. Additionally, semantic segmentation of the 3D model is obtained without significant computation. This approach is evaluated on the Scannet dataset where we significantly outperform state-of-the-art baselines (deep multiview stereo followed by traditional TSDF fusion) both quantitatively and qualitatively. We compare our 3D semantic segmentation to prior methods that use a depth sensor since no previous work attempts the problem with only RGB input.

general 3D geometric
simple back-projected feature