Another is to extend our framework and instead represent H with a deep GP prior #REFR .

The performance of the HGP on task 2, is somewhat underwhelming especially for larger K, relative to NODE.

We believe some of this effect can be attributed to our choice of GP prior.

We use a stationary GP prior, which is likely non-optimal for most Hamiltonian systems, which are typically nonstationary.

The assumption of non-stationarity likely leads to the poor generalisation at new phase space points, since the GP prior will revert to zero mean functions there, which is not the case for the NN models.

One option to improve the suitability of the GP prior is via the use of specifically designed kernels to represent symmetries in the system #OTHEREFR , or kernel structure learning #OTHEREFR .

Deep GPs are adept at modelling complex, nonstationary functions, and so would be well suited to the task.

Using the SVI scheme proposed by #OTHEREFR would make integration of deep GPs into our framework relatively straight forward, and would be interesting to undertake as part of future work.

The present method has significant potential at improving Bayesian online #OTHEREFR or policy-based RL (Yildiz et al., 2021) by incorporating Hamiltonian inductive biases.

This requires expanding the model towards Hamiltonian systems with external forces, which relax the energy conservation assumption.

Learning Energy Conserving Dynamics Efficiently With Hamiltonian Gaussian Processes

Abstract: Hamiltonian mechanics is one of the cornerstones of natural sciences. Recently there has been significant interest in learning Hamiltonian systems in a free-form way directly from trajectory data. Previous methods have tackled the problem of learning from many short, low-noise trajectories, but learning from a small number of long, noisy trajectories, whilst accounting for model uncertainty has not been addressed. In this work, we present a Gaussian process model for Hamiltonian systems with efficient decoupled parameterisation, and introduce an energy-conserving shooting method that allows robust inference from both short and long trajectories. We demonstrate the method's success in learning Hamiltonian systems in various data settings.

Deep Gaussian Processes

Abstract: In this paper we introduce deep Gaussian process (GP) models. Deep GPs are a deep belief network based on Gaussian process mappings. The data is modeled as the output of a multivariate GP. The inputs to that Gaussian process are then governed by another GP. A single layer model is equivalent to a standard GP or the GP latent variable model (GP-LVM). We perform inference in the model by approximate variational marginalization. This results in a strict lower bound on the marginal likelihood of the model which we use for model selection (number of layers and nodes per layer). Deep belief networks are typically applied to relatively large data sets using stochastic gradient descent for optimization. Our fully Bayesian treatment allows for the application of deep models even when data is scarce. Model selection by our variational bound shows that a five layer hierarchy is justified even when modelling a digit data set containing only 150 examples.


As discussed in #REFR , running time is important but hard to compare since it is affected by many factors.

We report the clock time for solving all the test instances in Table 1 .

For the UTSP (our method) and the state-of-the-art learning-based method Att-GCRN #OTHEREFR , we run the search algorithm on exactly the same environment (one Intel Xeon Gold 6326) for a fair comparison.

And for other baselines, we directly refer to the results from #OTHEREFR .

So the time there are only for indicative purpose since the computing hardware is not the same.

Unsupervised Learning For Solving The Travelling Salesman Problem

Abstract: We propose UTSP, an unsupervised learning (UL) framework for solving the Travelling Salesman Problem (TSP). We train a Graph Neural Network (GNN) using a surrogate loss. The GNN outputs a heat map representing the probability for each edge to be part of the optimal path. We then apply local search to generate our final prediction based on the heat map. Our loss function consists of two parts: one pushes the model to find the shortest path and the other serves as a surrogate for the constraint that the route should form a Hamiltonian Cycle. Experimental results show that UTSP outperforms the existing data-driven TSP heuristics. Our approach is parameter efficient as well as data efficient: the model takes ∼ 10% of the number of parameters and ∼ 0.2% of training samples compared with reinforcement learning or supervised learning methods.

Attention, Learn To Solve Routing Problems!

Abstract: The recently presented idea to learn heuristics for combinatorial optimization problems is promising as it can save costly development. However, to push this idea towards practical implementation, we need better models and better ways of training. We contribute in both directions: we propose a model based on attention layers with benefits over the Pointer Network and we show how to train this model using REINFORCE with a simple baseline based on a deterministic greedy rollout, which we find is more efficient than using a value function. We significantly improve over recent learned heuristics for the Travelling Salesman Problem (TSP), getting close to optimal results for problems up to 100 nodes. With the same hyperparameters, we learn strong heuristics for two variants of the Vehicle Routing Problem (VRP), the Orienteering Problem (OP) and (a stochastic variant of) the Prize Collecting TSP (PCTSP), outperforming a wide range of baselines and getting results close to highly optimized and specialized algorithms.


This bounding box is subdivided and serves as initial mesh, which is refined by an iterative mesh refinement as proposed in #REFR .

Transportation logistics and warehousing are a central part of every supply chain and play an important strategic role in the Industry 4.0 era #OTHEREFR .

However, several challenges need to be faced by companies working in the logistics sector: clients demand cheaper, faster and more pre- Figure 1 .

We take an RGB image as input and use Cube R-CNN's Cube Head #OTHEREFR to estimate a 3D bounding box.

For training and evaluation we present Parcel3D, a novel dataset of normal and damaged parcels with full 3D annotations.

cisely scheduled deliveries while at the same time, cities and highways are congested, and environmental concerns are of rising importance.

To tackle these challenges, process automation has huge potential #OTHEREFR .

Key processes for automation in logistics are identification, digital measurement, damage detection and tampering recognition of packaging units, all of which we work towards with the approach presented in this work. Identification is necessary for process documentation and parcel tracking.

Damage and tampering detection can be utilized to increase the safety and security along the supply chain #OTHEREFR .

Parcel3D: Shape Reconstruction From Single Rgb Images For Applications In Transportation Logistics

Abstract: We focus on enabling damage and tampering detection in logistics and tackle the problem of 3D shape reconstruction of potentially damaged parcels. As input we utilize single RGB images, which corresponds to use-cases where only simple handheld devices are available, e.g. for postmen during delivery or clients on delivery. We present a novel synthetic dataset, named Parcel3D, that is based on the Google Scanned Objects (GSO) dataset and consists of more than 13,000 images of parcels with full 3D annotations. The dataset contains intact, i.e. cuboid-shaped, parcels and damaged parcels, which were generated in simulations. We work towards detecting mishandling of parcels by presenting a novel architecture called CubeRefine R-CNN, which combines estimating a 3D bounding box with an iterative mesh refinement. We benchmark our approach on Parcel3D and an existing dataset of cuboidshaped parcels in real-world scenarios. Our results show, that while training on Parcel3D enables transfer to the real world, enabling reliable deployment in real-world scenarios is still challenging. CubeRefine R-CNN yields competitive performance in terms of Mesh AP and is the only model that directly enables deformation assessment by 3D mesh comparison and tampering detection by comparing viewpoint invariant parcel side surface representations. Dataset and code are available at https://a-nau.github.io/parcel3d.

Pixel2Mesh: Generating 3D Mesh Models From Single Rgb Images

Abstract: We propose an end-to-end deep learning architecture that produces a 3D shape in triangular mesh from a single color image. Limited by the nature of deep neural network, previous methods usually represent a 3D shape in volume or point cloud, and it is non-trivial to convert them to the more ready-to-use mesh model. Unlike the existing methods, our network represents 3D mesh in a graph-based convolutional neural network and produces correct geometry by progressively deforming an ellipsoid, leveraging perceptual features extracted from the input image. We adopt a coarse-to-fine strategy to make the whole deformation procedure stable, and define various of mesh related losses to capture properties of different levels to guarantee visually appealing and physically accurate 3D geometry. Extensive experiments show that our method not only qualitatively produces mesh model with better details, but also achieves higher 3D shape estimation accuracy compared to the state-of-the-art.


Compared with recent DIL #REFR , our method still achieves better results on MARS and DukeMCMT-VID.

Meanwhile, with pyramid spatial-temporal learning, PSTA #OTHEREFR gains remarkable 91.5% and 98.3% Rank-1 accuracy on MARS and DukeMCMT-VID datasets.

Different from PSTA, we propose a novel spatialtemporal complementary learning framework to obtain richer video representations.

Thus, we obtain better performance than PSTA on MARS and DukeMCMT-VID, which validates the effectiveness of our proposed method.

When compared with some methods that utilize Transformer, our method also shows great superiority in performance.

For example, compared with STT #OTHEREFR , our method has 1.2% and 3.6% gains in mAP and Rank-1 accuracy.

Besides, TMT #OTHEREFR introduces a multi-view Transformer to extract comprehensive video representations. However, our method still performs better than TMT.

We note that all these methods add Transformer layers after CNN backbones to obtain enhanced representations.

Different from them, we deeply couple CNNs and Transformers, and propose a complementary content attention for spatial complementary learning.

Besides, the hierarchical temporal aggregation is leveraged to progressively integrate temporal information.

In this way, we utilize two kinds of typical visual features from the same videos and achieve spatial-temporal complementary learning for more informative representations.

Deeply-Coupled Convolution-Transformer With Spatial-Temporal Complementary Learning For Video-Based Person Re-Identification

Abstract: Advanced deep Convolutional Neural Networks (CNNs) have shown great success in video-based person Re-Identification (Re-ID). However, they usually focus on the most obvious regions of persons with a limited global representation ability. Recently, it witnesses that Transformers explore the inter-patch relations with global observations for performance improvements. In this work, we take both sides and propose a novel spatial-temporal complementary learning framework named Deeply-Coupled Convolution-Transformer (DCCT) for high-performance video-based person Re-ID. Firstly, we couple CNNs and Transformers to extract two kinds of visual features and experimentally verify their complementarity. Further, in spatial, we propose a Complementary Content Attention (CCA) to take advantages of the coupled structure and guide independent features for spatial complementary learning. In temporal, a Hierarchical Temporal Aggregation (HTA) is proposed to progressively capture the inter-frame dependencies and encode temporal information. Besides, a gated attention is utilized to deliver aggregated temporal information into the CNN and Transformer branches for temporal complementary learning. Finally, we introduce a self-distillation training strategy to transfer the superior spatial-temporal knowledge to backbone networks for higher accuracy and more efficiency. In this way, two kinds of typical features from same videos are integrated mechanically for more informative representations. Extensive experiments on four public Re-ID benchmarks demonstrate that our framework could attain better performances than most state-of-the-art methods.

Few-Shot Deep Adversarial Learning For Video-Based Person Re-Identification

Abstract: Video-based person re-identification (re-ID) refers to matching people across camera views from arbitrary unaligned video footages. Existing methods rely on supervision signals to optimise a projected space under which the distances between inter/intra-videos are maximised/minimised. However, this demands exhaustively labelling people across camera views, rendering them unable to be scaled in large networked cameras. Also, it is noticed that learning effective video representations with view invariance is not explicitly addressed for which features exhibit different distributions otherwise. Thus, matching videos for person re-ID demands flexible models to capture the dynamics in time-series observations and learn view-invariant representations with access to limited labeled training samples. In this paper, we propose a novel few-shot deep learning approach to video-based person re-ID, to learn comparable representations that are discriminative and view-invariant. The proposed method is developed on the variational recurrent neural networks (VRNNs) and trained adversarially to produce latent variables with temporal dependencies that are highly discriminative yet view-invariant in matching persons. Through extensive experiments conducted on three benchmark datasets, we empirically show the capability of our method in creating view-invariant temporal features and state-of-the-art performance achieved by our method.


Recently, under the same capacity assumption, the work #REFR studies the mean square convergence of averaging online estimator with least square loss and multiple passes.

A small value of β implies a fast polynomially decaying rate at least achieved by the eigenvalues {λ k } k∈N .

One can refer to Theorem 5 in our recent work #OTHEREFR , which provides a characterization of the relationship between the capacity assumption (9) in this paper and the decaying rate of integral operator eigenvalues.

If the eigenvalues decay exponentially, the index β can be arbitrarily close to zero.

The polynomially decaying of the eigenvalue is typical for the Sobolev smooth kernels on domains in Euclidean spaces, and the parameter β depends on the smoothness of the kernel K.

While the exponentially decaying integral operator is typical for the analytic kernels on domains in Euclidean spaces.

Under the regularity condition (7) on the target function f ρ and the capacity assumption (9) on the hypothesis space H K , we obtain the following sharp capacity dependent results for strong convergence in H K .

t=1 be defined by algorithm #OTHEREFR with a windowing function W .

Assume that W satisfies (2) and (3) with some p > 0, the regularity condition #OTHEREFR holds with r > 1 2 and the capacity assumption (9) holds with 0 < β < 1. Choose step size η = 1

η 0 T 1−2r−β 2r+β with η 0 ≥ max C W κ 2 , 1 e + 2κ 2 W ′ + (0) 2 .

Optimality Of Robust Online Learning †

Abstract: In this paper, we study an online learning algorithm with a robust loss function L σ for regression over a reproducing kernel Hilbert space (RKHS). The loss function L σ involving a scaling parameter σ > 0 can cover a wide range of commonly used robust losses. The proposed algorithm is then a robust alternative for online least squares regression aiming to estimate the conditional mean function. For properly chosen σ and step size, we show that the last iterate of this online algorithm can achieve optimal capacity independent convergence in the mean square distance. Moreover, if additional information on the underlying function space is known, we also establish optimal capacity dependent rates for strong convergence in RKHS. To the best of our knowledge, both of the two results are new to the existing literature of online learning.

Statistical Optimality Of Stochastic Gradient Descent On Hard Learning Problems Through Multiple Passes

Abstract: We consider stochastic gradient descent (SGD) for least-squares regression with potentially several passes over the data. While several passes have been widely reported to perform practically better in terms of predictive performance on unseen data, the existing theoretical analysis of SGD suggests that a single pass is statistically optimal. While this is true for low-dimensional easy problems, we show that for hard problems, multiple passes lead to statistically optimal predictions while single pass does not; we also show that in these hard models, the optimal number of passes over the data increases with sample size. In order to define the notion of hardness and show that our predictive performances are optimal, we consider potentially infinite-dimensional models and notions typically associated to kernel methods, namely, the decay of eigenvalues of the covariance matrix of the features and the complexity of the optimal predictor as measured through the covariance matrix. We illustrate our results on synthetic experiments with non-linear kernel methods and on a classical benchmark with a linear model.


Unmanned aerial vehicle (UAV), thanks to its low cost, high flexibility and high probability of line-of-propagation (LoP) links, has become an attractive means of improving air-ground transmission quality #REFR .

Nowadays, wireless networks serve a wide range of civilian and military applications and have become an essential part of our routine #OTHEREFR , #OTHEREFR .

Improving the performance of wireless communication has become a popular research topic.

In general, UAVs can not only act as base stations #OTHEREFR , relays #OTHEREFR , etc, to transmit or forward signals, but also for data collection #OTHEREFR and positioning #OTHEREFR . Currently, UAV transmission technology has been widely explored.

For instance, the authors in #OTHEREFR considered a cellular network deployment in which UAV-to-UAV launch-receive pairs used the identical spectrum as the uplink of cellular ground users, and the performances of underlay and overlay spectrum sharing mechanisms were analyzed and compared.

In #OTHEREFR , to enhance the throughput of a single-cell multiuser orthogonal frequency division multiple access network with single UAV while guaranteeing the user fairness, an efficient method was proposed that outperformed the random and cellular schemes in terms of user fairness and sum rate.

UAV-enabled relay under malicious jamming was considered in #OTHEREFR , and the successive convex approximation (SCA) algorithm was employed to maximize the end-to-end throughput.

However, UAV network also presents some challenges that affect their performance.

Joint Beamforming And Phase Shift Design For Hybrid-Irs-And-Uav-Aided Directional Modulation Network

Abstract: Recently, intelligent reflecting surface (IRS) and unmanned aerial vehicle (UAV) have been introduced into wireless communication systems to enhance the performance of air-ground transmission. To make a good balance between performance, cost, and power consumption, a hybrid-IRS-and-UAV-assisted directional modulation (DM) network is investigated in this paper, where the hybrid IRS consists of passive and active reflecting elements. To maximize the achievable rate, three optimization algorithms, called maximum signal-to-noise ratio (SNR)-fractional programming (FP) (Max-SNR-FP), maximum SNR-equal amplitude reflecting (EAR) (Max-SNR-EAR), and maximum SNR-majorization-minimization (MM) (Max-SNR-MM), are proposed to jointly design the beamforming vector and phase shift matrix (PSM) of hybrid IRS by alternately optimizing one and giving another. The Max-SNR-FP method employs the successive convex approximation and FP methods to derive the beamforming vector and hybrid IRS PSM. The Max-SNR-EAR method adopts the maximum signal-to-leakagenoise ratio method and the criteria of phase alignment and EAR to design them. In addition, the Max-SNR-MM method utilizes the MM criterion to derive the IRS PSM. Simulation results show that the rates harvested by the proposed three methods are slightly lower than those of active IRS with higher power consumption, which are 35 percent higher than those of no IRS and random phase IRS, while passive IRS achieves only about 17 percent rate gain over the latter. Moreover, compared to Max-SNR-FP, the proposed Max-SNR-EAR and Max-SNR-MM methods make an obvious complexity degradation at the price of a slight performance loss.

Wireless Communications With Unmanned Aerial Vehicles: Opportunities And Challenges

Abstract: Wireless communication systems that include unmanned aerial vehicles promise to provide cost-effective wireless connectivity for devices without infrastructure coverage. Compared to terrestrial communications or those based on high-altitude platforms, on-demand wireless systems with low-altitude UAVs are in general faster to deploy, more flexibly reconfigured, and likely to have better communication channels due to the presence of short-range line-of-sight links. However, the utilization of highly mobile and energy-constrained UAVs for wireless communications also introduces many new challenges. In this article, we provide an overview of UAV-aided wireless communications, by introducing the basic networking architecture and main channel characteristics, highlighting the key design considerations as well as the new opportunities to be exploited.


Experiment 5: T-Sne Visualization Of Generated Data

Similar to the previous dataset, the DPWGAN #REFR produces a distribution with an offset, resulting in a bad performance.

between the private and public data, which is reflected in the accuracy, as the public classifier cannot perform well on the private dataset. Furthermore, the two classes are not separated well.

In contrast to that, Figure 1d shows that the generated data of the GSWGAN #OTHEREFR , although it is different from the original distribution, separated the two classes well.

In addition, this model was able to perform well, although the distribution shows some differences.

For the accuracy results related to this figure, the reader is referred to Table 2 .

The last dataset is visualized in Figure 1g and Figure 1h .

In contrast to that, the GSWGAN #OTHEREFR generated a dataset that shows a similar distribution within the T-SNE plot, resulting in high performance for the classifier trained on that dataset.

One general finding across all datasets was that the DPWGAN #OTHEREFR produces data that is less similar to the original data.

This can be explained by the fact that differential privacy is applied to both the discriminator and generator, resulting in a worse performance of the discriminator.

This is not the case for the GSWGAN #OTHEREFR as the privacy constraints are only applied to the generator.

The stronger discriminator seems to improve the quality of the generated data.

From Private To Public: Benchmarking Gans In The Context Of Private Time Series Classification

Abstract: Deep learning has proven to be successful in various domains and for different tasks. However, when it comes to private data several restrictions are making it difficult to use deep learning approaches in these application fields. Recent approaches try to generate data privately instead of applying a privacy-preserving mechanism directly, on top of the classifier. The solution is to create public data from private data in a manner that preserves the privacy of the data. In this work, two very prominent GAN-based architectures were evaluated in the context of private time series classification. In contrast to previous work, mostly limited to the image domain, the scope of this benchmark was the time series domain. The experiments show that especially GSWGAN performs well across a variety of public datasets outperforming the competitor DPWGAN. An analysis of the generated datasets further validates the superiority of GSWGAN in the context of time series generation.

Differentially Private Generative Adversarial Network

Abstract: Generative Adversarial Network (GAN) and its variants have recently attracted intensive research interests due to their elegant theoretical foundation and excellent empirical performance as generative models. These tools provide a promising direction in the studies where data availability is limited. One common issue in GANs is that the density of the learned generative distribution could concentrate on the training data points, meaning that they can easily remember training samples due to the high model complexity of deep networks. This becomes a major concern when GANs are applied to private or sensitive data such as patient medical records, and the concentration of distribution may divulge critical patient information. To address this issue, in this paper we propose a differentially private GAN (DPGAN) model, in which we achieve differential privacy in GANs by adding carefully designed noise to gradients during the learning procedure. We provide rigorous proof for the privacy guarantee, as well as comprehensive empirical evidence to support our analysis, where we demonstrate that our method can generate high quality data points at a reasonable privacy level.


The answers (yes/no) are marked by human annotators if certain Wikipedia pages contain sufficient information to address the questions. • OpenbookQA (OQA) #REFR is a multiple-choice reasoning dataset. The questions are derived from 1,326 core science facts.

We employ ten widely used benchmark QA datasets in our study, including BoolQ #OTHEREFR , Open-bookQA (OQA) #OTHEREFR , RACE #OTHEREFR , ARC #OTHEREFR , Common-senseQA (CQA) #OTHEREFR , SQuAD1 #OTHEREFR , SQuAD2 #OTHEREFR , Narra-tiveQA (NQA) #OTHEREFR , ELI5 #OTHEREFR , and TruthfulQA (TQA) #OTHEREFR .

These datasets encompass a broad range of QA capabilities, such as reading comprehension (BoolQ, SQuDA1/2, RACE), reasoning (OQA, ARC), commonsense (CQA), full document comprehension (NQA, ELI5), and truthfulness (TQA).

Furthermore, they comprise all four QA tasks #OTHEREFR , including yes/no (BoolQ), multiple-choice (OQA, RACE, ARC, CQA), extractive (SQuAD 1/2), and abstractive tasks (NQA, ELI5, TQA).

They thus offer a solid foundation to comprehensively evaluate the ChatGPT's reliability in various realworld QA scenarios.

Their details are outlined below and summarized in Table 2. • BoolQ #OTHEREFR is a yes/no reading comprehension dataset. The questions are derived from aggregated Google searches.

The answers consist of 4 candidates, of which only one is correct, requiring reasoning between questions and the given science facts and common knowledge.

• RACE #OTHEREFR is a multiple-choice reading comprehension dataset.

The questions are derived from English exams for Chinese students.

The answers include 4 candidates, of which only one is correct, requiring reading comprehension of English passages.

• ARC #OTHEREFR is a multiple-choice reasoning dataset.

In Chatgpt We Trust? Measuring And Characterizing The Reliability Of Chatgpt

Abstract: The way users acquire information is undergoing a paradigm shift with the advent of ChatGPT. Unlike conventional search engines, ChatGPT retrieves knowledge from the model itself and generates answers for users. ChatGPT's impressive question-answering (QA) capability has attracted more than 100 million users within a short period of time but has also raised concerns regarding its reliability. In this paper, we perform the first large-scale measurement of ChatGPT's reliability in the generic QA scenario with a carefully curated set of 5,695 questions across ten datasets and eight domains. We find that ChatGPT's reliability varies across different domains, especially underperforming in law and science questions. We also demonstrate that system roles, originally designed by OpenAI to allow users to steer ChatGPT's behavior, can impact ChatGPT's reliability. We further show that ChatGPT is vulnerable to adversarial examples, and even a single character change can negatively affect its reliability in certain cases. We believe that our study provides valuable insights into ChatGPT's reliability and underscores the need for strengthening the reliability and security of large language models (LLMs).

Can A Suit Of Armor Conduct Electricity? A New Dataset For Open Book Question Answering

Abstract: We present a new kind of question answering dataset, OpenBookQA, modeled after open book exams for assessing human understanding of a subject. The open book that comes with our questions is a set of 1329 elementary level science facts. Roughly 6000 questions probe an understanding of these facts and their application to novel situations. This requires combining an open book fact (e.g., metals conduct electricity) with broad common knowledge (e.g., a suit of armor is made of metal) obtained from other sources. While existing QA datasets over documents or knowledge bases, being generally self-contained, focus on linguistic understanding, OpenBookQA probes a deeper understanding of both the topic---in the context of common knowledge---and the language it is expressed in. Human performance on OpenBookQA is close to 92%, but many state-of-the-art pre-trained QA methods perform surprisingly poorly, worse than several simple neural baselines we develop. Our oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts. We leave it as a challenge to solve the retrieval problem in this multi-hop setting and to close the large gap to human performance.


Then, the relation between response range and grazing angle could be described in Equation #REFR .

However, in some case it is observed that the response is very scattered, and the response peak is often indistinguishable from other non-response points.

Figure 3 helps to understand the relation between the response range and grazing angle.

The black dotted line denotes the detected vectorized lane (hereinafter referred to as "vector line"), and the green dotted lines alongside indicate the range of pixels that generate a response on the heatmap.

The greener the background is, the more positive the sample is. d is the radius of the range.

The tangents of two guide lines are illustrated in orange and magenta, with grazing angles as α1 and α2, respectively.

The bigger the angle (close to 90 • ), the tighter the response range.

As the grazing angle gets smaller, the range scatters rapidly. This reflects the results in Figure 1a .

Therefore, the guide line principle is to "reduce the number of lanes of small grazing angles".

Canet: Curved Guide Line Network With Adaptive Decoder For Lane Detection

Abstract: Lane detection is challenging due to the complicated onroad scenarios and line deformation from different camera perspectives. Lots of solutions were proposed, but can not deal with "corner lanes" well. To address this problem, this paper proposes a new top-down deep learning lane detection approach, CANET. A lane instance is first responded by the heatmap on the U-shaped "curved guide line" at global semantic level, thus the corresponding features of each lane are aggregated at the response point. Then CANET obtains the heatmap response of the entire lane through conditional convolution, and finally decodes the point set to describe lanes via adaptive decoder. The prototype is implemented with Pytorch, and evaluated against 3 well-known datasets extensively. The experimental results show that CANET reaches SOTA in different metrics. Our code will be released soon.

Yolact: Real-Time Instance Segmentation

Abstract: We present a simple, fully-convolutional model for real-time instance segmentation that achieves 29.8 mAP on MS COCO at 33.5 fps evaluated on a single Titan Xp, which is significantly faster than any previous competitive approach. Moreover, we obtain this result after training on only one GPU. We accomplish this by breaking instance segmentation into two parallel subtasks: (1) generating a set of prototype masks and (2) predicting per-instance mask coefficients. Then we produce instance masks by linearly combining the prototypes with the mask coefficients. We find that because this process doesn't depend on repooling, this approach produces very high-quality masks and exhibits temporal stability for free. Furthermore, we analyze the emergent behavior of our prototypes and show they learn to localize instances on their own in a translation variant manner, despite being fully-convolutional. Finally, we also propose Fast NMS, a drop-in 12 ms faster replacement for standard NMS that only has a marginal performance penalty.


Fortunately, several privacy-preserving distributed learning frameworks, such as FL #REFR , have been proposed to empower privacypreserving AIGC model fine-tuning and inference at mobile AIGC networks.

For example, AI-generated text can be used by malicious users to complete phishing emails, thus compromising the security and privacy of normal users #OTHEREFR .

To ensure secure AIGC services, providers must choose trusted AIGC solutions and train AI models in a secure manner while providing secure hints and answers to AIGC service users.

1) Privacy-preserving AIGC Service Provisioning: During the lifecycle of providing AIGC services, privacy information in large-scale datasets and user requests needs to be kept secure to prevent privacy breaches.

In mobile AIGC networks, the generation and storage of data for AIGC model training occur at edge servers and mobile devices #OTHEREFR .

Unlike resourceful cloud data centers, edge and mobile layers have limited defense capacities against various attacks.

In preserving user privacy in AIGC networks, FL is a distributed ML approach that allows users to transmit local models instead of data during model training #OTHEREFR - #OTHEREFR . Specifically, as illustrated in Fig.

20 , there are two major approaches to employing FL in AIGC networks • Secure aggregation: While FL is being learned, the mobile devices send local updates to edge servers for global aggregation.

During global aggregation, authenticated encryption allows the use of secret sharing mechanisms.

• Differential privacy: Differential privacy can prevent FL servers from identifying the owners of a local update.

Differential privacy is similar to secure aggregation in that it prevents FL servers from identifying owners of local updates.

Unleashing The Power Of Edge-Cloud Generative Ai In Mobile Networks: A Survey Of Aigc Services

Abstract: Intelligence-Generated Content (AIGC) is an automated method for generating, manipulating, and modifying valuable and diverse data using AI algorithms creatively. This survey paper focuses on the deployment of AIGC applications, e.g., ChatGPT and Dall-E, at mobile edge networks, namely mobile AIGC networks, that provide personalized and customized AIGC services in real time while maintaining user privacy. We begin by introducing the background and fundamentals of generative models and the lifecycle of AIGC services at mobile AIGC networks, which includes data collection, training, finetuning, inference, and product management. We then discuss the collaborative cloud-edge-mobile infrastructure and technologies required to support AIGC services and enable users to access AIGC at mobile edge networks. Furthermore, we explore AIGCdriven creative applications and use cases for mobile AIGC networks. Additionally, we discuss the implementation, security, and privacy challenges of deploying mobile AIGC networks. Finally, we highlight some future research directions and open issues for the full realization of mobile AIGC networks.

Federated Learning In Mobile Edge Networks: A Comprehensive Survey

Abstract: In recent years, mobile devices are equipped with increasingly advanced sensing and computing capabilities. Coupled with advancements in Deep Learning (DL), this opens up countless possibilities for meaningful applications. Traditional Machine Learning (ML) approaches require the data to be centralized in a cloud server. However, this results in critical issues related to unacceptable latency and communication inefficiency. To this end, Mobile Edge Computing (MEC) has been proposed to bring intelligence closer to the edge, where data is produced. However, conventional enabling technologies for ML at mobile edge networks still require personal data to be shared with external parties, e.g., edge servers. Recently, in light of increasingly stringent data privacy legislation and growing privacy concerns, the concept of Federated Learning (FL) has been introduced. In FL, end devices use their local data to train an ML model required by the server. The end devices then send the model updates rather than raw data to the server for aggregation. FL can serve as an enabling technology in mobile edge networks since it enables the collaborative training of an ML model and also enables DL for mobile edge network optimization. However, in a large-scale and complex mobile edge network, heterogeneous devices with varying constraints are involved. This raises challenges of communication costs, resource allocation, and privacy and security in the implementation of FL at scale. In this survey, we begin with an introduction to the background and fundamentals of FL. Then, we highlight the aforementioned challenges of FL implementation and review existing solutions. Furthermore, we present the applications of FL for mobile edge network optimization. Finally, we discuss the important challenges, open issues and future research directions in FL.

privacypreserving AIGC model
mobile AIGC networks


EgoNeRF shows comparable results in the Synthetic-NeRF dataset #REFR , which contains 8 synthetic objects.

The spherical grid of EgoNeRF aligns nicely with outward-facing scenes, not inward-facing images of typical NeRF settings.

We optionally report results from widely-used datasets for novel view synthesis in Tab. 3. Table 5 . Out of distribution test.

r is the distance between the center of training camera trajectory and position of test view.

In mip-NeRF 360 dataset #OTHEREFR , which contains inward-facing objects but has unbounded background scenes, EgoNeRF outperforms other baselines except mip-NeRF 360.

Balanced Spherical Grid For Egocentric View Synthesis

Abstract: Figure 1. We propose a practical solution to reconstruct large-scale scenes from a short egocentric video. (a) Our scalable capturing setup observes the holistic environment by casually swiping a selfie stick with an omnidirectional camera attached. (b) Then we optimize our balanced spherical feature grids which are tailored for the outward-looking setup.

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.

Synthetic-NeRF dataset
8 synthetic objects


They show that an Inverse-Wishart distribution can be applied as the prior on the covariance, which results in a Student's t distribution over the noise #REFR .

We first review their methodology for uncorrelated noise prediction (i.e., θ(ψ * |D) = W(ψ * |D)), then present how we adapt their methodology. Vega-Brown et al.

#OTHEREFR formulate an estimator for θ * given a target feature, ψ * .

A joint posterior between θ * and θ 1:N is formed using Bayes' rule and θ 1:N is then marginalized out, resulting in

where we note the uncorrelated factorization of the likelihood terms.

In our work we will apply an uninformative prior, p(θ * |ψ * ) ∝ 1, which is an adaptation of their prior work, CELLO #OTHEREFR , #OTHEREFR .

#OTHEREFR is their choice in modelling the likelihood in (15) (referred to as the extended likelihood), as

where h(ψ i , ψ * ) is a kernel function.

Towards Consistent Batch State Estimation Using A Time-Correlated Measurement Noise Model

Abstract: In this paper, we present an algorithm for learning time-correlated measurement covariances for application in batch state estimation. We parameterize the inverse measurement covariance matrix to be block-banded, which conveniently factorizes and results in a computationally efficient approach for correlating measurements across the entire trajectory. We train our covariance model through supervised learning using the groundtruth trajectory. In applications where the measurements are time-correlated, we demonstrate improved performance in both the mean posterior estimate and the covariance (i.e., improved estimator consistency). We use an experimental dataset collected using a mobile robot equipped with a laser rangefinder to demonstrate the improvement in performance. We also verify estimator consistency in a controlled simulation using a statistical test over several trials.

Probe-Gk: Predictive Robust Estimation Using Generalized Kernels

Abstract: Many algorithms in computer vision and robotics make strong assumptions about uncertainty, and rely on the validity of these assumptions to produce accurate and consistent state estimates. In practice, dynamic environments may degrade sensor performance in predictable ways that cannot be captured with static uncertainty parameters. In this paper, we employ fast nonparametric Bayesian inference techniques to more accurately model sensor uncertainty. By setting a prior on observation uncertainty, we derive a predictive robust estimator, and show how our model can be learned from sample images, both with and without knowledge of the motion used to generate the data. We validate our approach through Monte Carlo simulations, and report significant improvements in localization accuracy relative to a fixed noise model in several settings, including on synthetic data, the KITTI dataset, and our own experimental platform.


For a given algorithm, varying the threshold values can result in distinct performance profiles, e.g., a precision-recall curve, and several thresholds may be used together for the purposes of evaluation and comparison, as is common practice in the object detection literature #REFR .

We provide Segment-to-text IoU (IoU st ) scores with several τ CLIP threshold values in Figure 12 and Table 6 .

Selecting the threshold τ CLIP is more challenging, since there is no established consensus or user studies to rely on.

Figure 13 shows histograms of CLIP similarity scores between groundtruth image segments and their corresponding ground-truth labels in Pascal Context and Pascal VOC datasets.

Given the distributions, we select τ CLIP = 0.1 to be on the safe side to report Segment-to-text IoU scores in the main experiment.

It is important to note that for our zero-guidance segmentation problem, the thresholds τ CLIP and τ SBERT are used in the label reassignment verification process (Section 4.2), which is part of the evaluation not the segmentation algorithm itself.

We use τ IoU = 0.5, which is commonly used in object detection tasks to determine if a predicted bounding box is "correct" compared to the ground

Zero-Guidance Segmentation Using Zero Segment Labels

Abstract: CLIP has enabled new and exciting joint vision-language applications, one of which is open-vocabulary segmentation, which can locate any segment given an arbitrary text query. In our research, we ask whether it is possible to discover semantic segments without any user guidance in the form of text queries or predefined classes, and label them using natural language automatically? We propose a novel problem zero-guidance segmentation and the first baseline that leverages two pre-trained generalist models, DINO and CLIP, to solve this problem without any finetuning or segmentation dataset. The general idea is to first segment an image into small over-segments, encode them into CLIP's visual-language space, translate them into text labels, and merge semantically similar segments together. The key challenge, however, is how to encode a visual segment into a segment-specific embedding that balances global and local context information, both useful for recognition. Our main contribution is a novel attention-masking technique that balances the two contexts by analyzing the attention layers inside CLIP. We also introduce several metrics for the evaluation of this new task. With CLIP's innate knowledge, our method can precisely locate the Mona Lisa painting among a museum crowd (Figure 1

Object Detection In 20 Years: A Survey

Abstract: Object detection, as of one the most fundamental and challenging problems in computer vision, has received great attention in recent years. Its development in the past two decades can be regarded as an epitome of computer vision history. If we think of today's object detection as a technical aesthetics under the power of deep learning, then turning back the clock 20 years we would witness the wisdom of cold weapon era. This paper extensively reviews 400+ papers of object detection in the light of its technical evolution, spanning over a quarter-century's time (from the 1990s to 2019). A number of topics have been covered in this paper, including the milestone detectors in history, detection datasets, metrics, fundamental building blocks of the detection system, speed up techniques, and the recent state of the art detection methods. This paper also reviews some important detection applications, such as pedestrian detection, face detection, text detection, etc, and makes an in-deep analysis of their challenges as well as technical improvements in recent years.


To process the masked image, we resort to the partial convolution operator (PConv) #REFR , whose convolution kernel is renormalized to be applied only on unmasked pixels.

Unlike prior work #OTHEREFR whose ultimate goal is to localize the forgery mask, we reuse the forgery mask to help HiFi-Net learn the optimal feature for classifying fine-grained forged attributes.

Specifically, we generate a binary maskM, then overlayM with the input image as X M to obtain the masked image

The idea is to have feature maps only describe pixels at the manipulated region.

PConv acts as conditioned dot product for each kernel, conditioned on the mask. Denoting W par as the convolution kernel, we have:

where the dot product is "renormalized" to account for zeros in the mask.

At different layers, we update and propa- gate the new maskM according to the following equation:

Hierarchical Fine-Grained Image Forgery Detection And Localization

Abstract: Differences in forgery attributes of images generated in CNN-synthesized and image-editing domains are large, and such differences make a unified image forgery detection and localization (IFDL) challenging. To this end, we present a hierarchical fine-grained formulation for IFDL representation learning. Specifically, we first represent forgery attributes of a manipulated image with multiple labels at different levels. Then we perform fine-grained classification at these levels using the hierarchical dependency between them. As a result, the algorithm is encouraged to learn both comprehensive features and inherent hierarchical nature of different forgery attributes, thereby improving the IFDL representation. Our proposed IFDL framework contains three components: multi-branch feature extractor, localization and classification modules. Each branch of the feature extractor learns to classify forgery attributes at one level, while localization and classification modules segment the pixel-level forgery region and detect imagelevel forgery, respectively. Lastly, we construct a hierarchical fine-grained dataset to facilitate our study. We demonstrate the effectiveness of our method on 7 different benchmarks, for both tasks of IFDL and forgery attribute classification. Our source code and dataset can be found: github.com/CHELSEA234/HiFi-IFDL.

Image Inpainting For Irregular Holes Using Partial Convolutions

Abstract: Existing deep learning based image inpainting methods use a standard convolutional network over the corrupted image, using convolutional filter responses conditioned on both valid pixels as well as the substitute values in the masked holes (typically the mean value). This often leads to artifacts such as color discrepancy and blurriness. Post-processing is usually used to reduce such artifacts, but are expensive and may fail. We propose the use of partial convolutions, where the convolution is masked and renormalized to be conditioned on only valid pixels. We further include a mechanism to automatically generate an updated mask for the next layer as part of the forward pass. Our model outperforms other methods for irregular masks. We show qualitative and quantitative comparisons with other methods to validate our approach.


The authors in #REFR propose DNN architecture for solving the multilabel classification task, which incorporates the construction of label embeddings with feature and label interdependency awareness.

The work #OTHEREFR examines the same problem statement of multi-label classification in an event stream as we do.

The authors' model targets capturing temporal and probabilistic dependencies between concurrent event types by encoding historical information with a transformer and then leveraging a conditional mixture of Bernoulli experts.

Approaches to leveraging label dependencies.

The authors of #OTHEREFR construct a model called C-Tran for a multi-label image classification task that leverages Transformer architecture that encourages capturing the dependencies among image features and target labels.

The key idea is to train the model with label masking.

A label-correlation sensitive loss improves the efficiency of the constructed model.

Another popular way to consider label relationships is to use Graph Neural Networks as a part of the pipeline.

Namely, #OTHEREFR captures the correlation between the labels in the task of Multi-Label Text Classification by adopting Graph Attention Network (GAT).

They predict the final set of labels combining feature vectors from BiLSTM and attended label features from GAT.

Event sequences processing also tries to derive dependencies between different event types and consider specific attention mechanisms #OTHEREFR .

Label Attention Network For Sequential Multi-Label Classification: You Were Looking At A Wrong Self-Attention

Abstract: Most of the available user information can be represented as a sequence of timestamped events. Each event is assigned a set of categorical labels whose future structure is of great interest. For instance, our goal is to predict a group of items in the next customer's purchase or tomorrow's client transactions. This is a multi-label classification problem for sequential data. Modern approaches focus on transformer architecture for sequential data introducing self-attention for the elements in a sequence. In that case, we take into account events' time interactions but lose information on label inter-dependencies. Motivated by this shortcoming, we propose leveraging a self-attention mechanism over labels preceding the predicted step. As our approach is a Label-Attention NETwork, we call it LANET. Experimental evidence suggests that LANET outperforms the established models' performance and greatly captures interconnections between labels. For example, the micro-AUC of our approach is 0.9536 compared to 0.7501 for a vanilla transformer. We provide an implementation of LANET to facilitate its wider usage.

Learning Deep Latent Spaces For Multi-Label Classification

Abstract: Multi-label classification is a practical yet challenging task in machine learning related fields, since it requires the prediction of more than one label category for each input instance. We propose a novel deep neural networks (DNN) based model, Canonical Correlated AutoEncoder (C2AE), for solving this task. Aiming at better relating feature and label domain data for improved classification, we uniquely perform joint feature and label embedding by deriving a deep latent space, followed by the introduction of label-correlation sensitive loss function for recovering the predicted label outputs. Our C2AE is achieved by integrating the DNN architectures of canonical correlation analysis and autoencoder, which allows end-to-end learning and prediction with the ability to exploit label dependency. Moreover, our C2AE can be easily extended to address the learning problem with missing labels. Our experiments on multiple datasets with different scales confirm the effectiveness and robustness of our proposed method, which is shown to perform favorably against state-of-the-art methods for multi-label classification.

multilabel classification task
label embeddings


We will achieve this goal by designing a method for finding a nontrivial pair of subsets having equal sum and then, like in #REFR , applying the algorithm recursively to obtain 4, 8, 16, etc. disjoint subsets with equal sum.

In this section, we assume that our input is a sequence of vectors from Z n p .

We also assume that p is an odd prime as for p = 2 a zero sum subsequence can be obtained from n + 1 vectors in the form of a zero linear combination.

As subsequences can be represented as subsets of the index set, it will not be too misleading to use the term (sub)set for a (sub)sequence.

Our strategy will be finding p pairwise disjoint subsets of input vectors having equal sums.

Note that a pair of disjoint subsets with equal sum can be interpreted as a representation of the zero vector by a linear combination of the input vectors with nonzero coefficients 1 or −1 only.

Based on this, it will be convenient to use the term signed subsets and signed subset sums.

A signed subset of a set S of vectors is formally a function from S to the set {0, 1, −1}.

The support of such a signed subset is the set of elements on which the function takes nonzero values.

With some sloppiness, we use the term signed subset sum to refer both to the signed subset and to the value of the signed sum.

Zero Sum Subsequences And Hidden Subgroups

Abstract: We propose a method for solving the hidden subgroup problem in nilpotent groups. The main idea is iteratively transforming the hidden subgroup to its images in the quotient groups by the members of a central series, eventually to its image in the commutative quotient of the original group; and then using an abelian hidden subgroup algorithm to determine this image. Knowing this image allows one to descend to a proper subgroup unless the hidden subgroup is the full group. The transformation relies on finding zero sum subsequences of sufficiently large sequences of vectors over finite prime fields. We present a new deterministic polynomial time algorithm for the latter problem in the case when the size of the field is constant. The consequence is a polynomial time exact quantum algorithm for the hidden subgroup problem in nilpotent groups having constant nilpotency class and whose order only have prime factors also bounded by a constant.

On Solving Systems Of Diagonal Polynomial Equations Over Finite Fields

Abstract: We present an algorithm to solve a system of diagonal polynomial equations over finite fields when the number of variables is greater than some fixed polynomial of the number of equations whose degree depends only on the degree of the polynomial equations. Our algorithm works in time polynomial in the number of equations and the logarithm of the size of the field, whenever the degree of the polynomial equations is constant. As a consequence we design polynomial time quantum algorithms for two algebraic hidden structure problems: for the hidden subgroup problem in certain semidirect product p-groups of constant nilpotency class, and for the multi-dimensional univariate hidden polynomial graph problem when the degree of the polynomials is constant.


The encoder E is based on the StyleGAN2 #REFR discriminator architecture and compresses the input image I rgb to a vector v of dimension 512.

This RGB texture allows us to explicitly save information about high-frequency details and original colors, which are hard to preserve when mapping the whole image to a vector of limited dimensionality (as discussed below).

We additionally apply inpainting of small gaps with averaging neighbor pixels to fill the gaps in T rgb .

We also save the binary map of sampled pixels to the B smp and the map of the sampled and inpainted pixels to the B fill .

The main part of the neural texture is T gen .

It has the number of channels L = 16 and is generated using the encoder-generator architecture T gen = G(E(I rgb )).

The generator G( v) has the architecture of the StyleGAN2 generator and converts the vector v into a T gen neural texture with the number of channels L = 16 as in StylePeople #OTHEREFR .

The final neural texture used in our method has a dimension of 256 × 256 × 21 and consists of the concatenation of: the generated texture T gen (256 × 256 × 16), the texture T rgb sampled from the RGB image (256 × 256 × 3) and the two binary segmentation maps (B smp and B fill ):

We note that such an approach with the explicit use of RGB channels as part of the neural texture was originally used in #OTHEREFR .

We use the neural renderer θ(R(F UV , T )) to translate the rasterized image R(F UV , T ) with L channels into I rend output RGB image.

Dinar: Diffusion Inpainting Of Neural Textures For One-Shot Human Avatars

Abstract: We present DINAR, an approach for creating realistic rigged fullbody avatars from single RGB images. Similarly to previous works, our method uses neural textures combined with the SMPL-X body model to achieve photorealistic quality of avatars while keeping them easy to animate and fast to infer. To restore the texture, we use a latent diffusion model and show how such model can be trained in the neural texture space. The use of the diffusion model allows us to realistically reconstruct large unseen regions such as the back of a person given the frontal view. The models in our pipeline are trained using 2D images and videos only. In the experiments, our approach achieves state-of-the-art rendering quality and good generalization to new poses and viewpoints. In particular, the approach improves state-of-the-art on the SnapshotPeople public benchmark.

Analyzing And Improving The Image Quality Of Stylegan

Abstract: The style-based GAN architecture (StyleGAN) yields state-of-the-art results in data-driven unconditional generative image modeling. We expose and analyze several of its characteristic artifacts, and propose changes in both model architecture and training methods to address them. In particular, we redesign the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images. In addition to improving image quality, this path length regularizer yields the additional benefit that the generator becomes significantly easier to invert. This makes it possible to reliably attribute a generated image to a particular network. We furthermore visualize how well the generator utilizes its output resolution, and identify a capacity problem, motivating us to train larger models for additional quality improvements. Overall, our improved model redefines the state of the art in unconditional image modeling, both in terms of existing distribution quality metrics as well as perceived image quality.


Smoothing As discussed in Section 3.2, computing the cardinalities on an unsmooth d-DNNF causes computational overhead as the algorithm needs to keep track on the set of variables during the traversal #REFR .

To ensure smoothness, we add designated nodes for each unsmooth ∨-node as seen in Figure 3 in a preprocessing step.

Hereby, for each missing variable v ∈ missingVariables Φ in a child subtree Φ of the ∨-node, we add a disjunction v ∨ ¬v resulting in the following smooth disjunction on the right side.

Φ∈children Φ → Φ∈children (Φ ∧ v∈missingVariables Φ v ∨ ¬v)

The added subformulas are tautologies as v ∨ ¬v ≡ holds for every variable v.

Hence, when adding smoothing nodes to a subformula Φ, the result Φ ∧ equivalent to Φ.

Exploiting D-Dnnfs For Repetitive Counting Queries On Feature Models

Abstract: Feature models are commonly used to specify the valid configurations of a product line. In industry, feature models are often complex due to a large number of features and constraints. Thus, a multitude of automated analyses have been proposed. Many of those rely on computing the number of valid configurations which typically depends on solving a #SAT problem, a computationally expensive operation. Further, most counting-based analyses require numerous #SAT computations on the same feature model. In particular, many analyses depend on multiple computations for evaluating the number of valid configurations that include certain features or conform to partial configurations. Instead of using expensive repetitive computations on highly similar formulas, we aim to improve the performance by reusing knowledge between these computations. In this work, we are the first to propose reusing d-DNNFs for performing efficient repetitive queries on features and partial configurations. Our empirical evaluation shows that our approach is up-to 8,300 times faster (99.99% CPU-time saved) than the state of the art of repetitively invoking #SAT solvers. Applying our tool ddnnife reduces runtimes from days to minutes compared to using #SAT solvers.

A Knowledge Compilation Map

Abstract: We propose a perspective on knowledge compilation which calls for analyzing different compilation approaches according to two key dimensions: the succinctness of the target compilation language, and the class of queries and transformations that the language supports in polytime. We then provide a knowledge compilation map, which analyzes a large number of existing target compilation languages according to their succinctness and their polytime transformations and queries. We argue that such analysis is necessary for placing new compilation approaches within the context of existing ones. We also go beyond classical, flat target compilation languages based on CNF and DNF, and consider a richer, nested class based on directed acyclic graphs (such as OBDDs), which we show to include a relatively large number of target compilation languages.


Owens and Efros #REFR also utilize this pretext task, but instead take temporal video frames and audio as input.

where the pseudo-label z i is a one-hot vector representing whether the inputs are matched.

Matching prediction is widely used for modeling audiovisual correspondence (AVC).

AVC was introduced by L 3 -Net #OTHEREFR , which uses a fused representation from audio and video to make a binary prediction of whether the audioimage pair is from the same video clip.

This strategy is adopted by AVE-Net #OTHEREFR by only using Euclidean distance alignment without fusion, leading to localization of the object that sounds within an image.

Also, they sample video frames from the same video to construct unaligned pairs.

This makes the pretext task more difficult and thus learns betteraligned representation and more accurate localization.

In order to achieve better audio-visual localization and separation, researchers have proposed more complex pretext tasks.

One such approach is the mix-and-separate method #OTHEREFR , which combines audio signals from different videos to create an input mixture with known constituent source signals.

The network is then trained to separate the audio sources based on corresponding video frames by predicting binary spectrogram masks for each sound source.

Self-Supervised Multimodal Learning: A Survey

Abstract: Multimodal learning, which aims to understand and analyze information from multiple modalities, has achieved substantial progress in the supervised regime in recent years. However, the heavy dependence on data paired with expensive human annotations impedes scaling up models. Meanwhile, given the availability of large-scale unannotated data in the wild, self-supervised learning has become an attractive strategy to alleviate the annotation bottleneck. Building on these two directions, self-supervised multimodal learning (SSML) provides ways to leverage supervision from raw multimodal data. In this survey, we provide a comprehensive review of the state-of-the-art in SSML, which we categorize along three orthogonal axes: objective functions, data alignment, and model architectures. These axes correspond to the inherent characteristics of self-supervised learning methods and multimodal data. Specifically, we classify training objectives into instance discrimination, clustering, and masked prediction categories. We also discuss multimodal input data pairing and alignment strategies during training. Finally, we review model architectures including the design of encoders, fusion modules, and decoders, which are essential components of SSML methods. We review downstream multimodal application tasks, reporting the concrete performance of the state-of-the-art image-text models and multimodal video models, and also review real-world applications of SSML algorithms in diverse fields such as healthcare, remote sensing, and machine translation. Finally, we discuss challenges and future directions for SSML. A collection of related resources can be found at: https://github.com/ys-zong/awesome-self-supervised-multimodal-learning.

Audio-Visual Scene Analysis With Self-Supervised Multisensory Features

Abstract: The thud of a bouncing ball, the onset of speech as lips open—when visual and audio events occur together, it suggests that there might be a common, underlying event that produced both signals. In this paper, we argue that the visual and audio components of a video signal should be modeled jointly using a fused multisensory representation. We propose to learn such a representation in a self-supervised way, by training a neural network to predict whether video frames and audio are temporally aligned. We use this learned representation for three applications: (a) sound source localization, i.e. visualizing the source of sound in a video; (b) audio-visual action recognition; and (c) on/off-screen audio source separation, e.g. removing the off-screen translator’s voice from a foreign official’s speech. Code, models, and video results are available on our webpage: http://andrewowens.com/multisensory.


We first evaluate the performance of motion-compensation methods on rotational sequences #REFR by measuring the Root Mean Square (RMS) of angular velocity errors, as presented in Tab. 3.

CMax #OTHEREFR exhibits good performance for the 3-DoF rotational motion of event cameras, with the lowest time complexity among the evaluated methods.

DMin #OTHEREFR extends CMax to high-dimensional feature spaces and applies entropy minimization to the projected events, resulting in an improvement in the performance of approximately 20%. However, DMin is computationally expensive.

To address this issue, approximate DMin uses a truncated kernel to balance performance and efficiency.

ST-PPP #OTHEREFR offers an alternative solution by employing a probabilistic model, achieving the highest performance among the evaluated methods, with a 39% improvement in the shapes sequence.

Then, we assess the performance of both deep learning and motion-compensation methods on the outdoor day 1 sequence by measuring the APRE, ARRE, and AEE metrics. As shown in Tab.

Event-Based Simultaneous Localization And Mapping: A Comprehensive Survey

Abstract: In recent decades, visual simultaneous localization and mapping (vSLAM) has gained significant interest in both academia and industry. It estimates camera motion and reconstructs the environment concurrently using visual sensors on a moving robot. However, conventional cameras are limited by hardware, including motion blur and low dynamic range, which can negatively impact performance in challenging scenarios like high-speed motion and high dynamic range illumination. Recent studies have demonstrated that event cameras, a new type of bio-inspired visual sensor, offer advantages such as high temporal resolution, dynamic range, low power consumption, and low latency. This paper presents a timely and comprehensive review of event-based vSLAM algorithms that exploit the benefits of asynchronous and irregular event streams for localization and mapping tasks. The review covers the working principle of event cameras and various event representations for preprocessing event data. It also categorizes event-based vSLAM methods into four main categories: feature-based, direct, motion-compensation, and deep learning methods, with detailed discussions and practical guidance for each approach. Furthermore, the paper evaluates the state-of-the-art methods on various benchmarks, highlighting current challenges and future opportunities in this emerging research area. A public repository will be maintained to keep track of the rapid developments in this field at https://github.com/kun150kun/ESLAM-survey.

The Event-Camera Dataset And Simulator: Event-Based Data For Pose Estimation, Visual Odometry, And Slam

Abstract: New vision sensors, such as the dynamic and active-pixel vision sensor (DAVIS), incorporate a conventional global-shutter camera and an event-based sensor in the same pixel array. These sensors have great potential for high-speed robotics and computer vision because they allow us to combine the benefits of conventional cameras with those of event-based sensors: low latency, high temporal resolution, and very high dynamic range. However, new algorithms are required to exploit the sensor characteristics and cope with its unconventional output, which consists of a stream of asynchronous brightness changes (called “events”) and synchronous grayscale frames. For this purpose, we present and release a collection of datasets captured with a DAVIS in a variety of synthetic and real environments, which we hope will motivate research on new algorithms for high-speed and high-dynamic-range robotics and computer-vision applications. In addition to global-shutter intensity images and asynchronous events, we provide inertial measurements and ground-truth camera poses from a motion-capture system. The latter allows comparing the pose accuracy of ego-motion estimation algorithms quantitatively. All the data are released both as standard text files and binary files (i.e. rosbag). This paper provides an overview of the available data and describes a simulator that we release open-source to create synthetic event-camera data.


Here, L(Y, X ) denotes the fidelity term and ψ(Z (k) ) represents the low-rank characterization of Z (k) (which denotes the k-th frontal (spatial) slice of Z ∈ R h×w×b #REFR ).

We first introduce the general formulation of t-SVD.

Suppose that the noisy HSI Y ∈ R h×w×b admits Y = X + N , where X denotes the clean HSI and N denotes noise.

To infer the underlying clean HSI X from the observed Y, t-SVD method generally formulates the following model:

φ θ (•) : R h×w×b → R h×w×b denotes a transform with learnable parameters θ, which transforms the low-rank representation Z into the original domain.

Sometimes the transform φ θ (•) may not be learnable (e.g., the fixed DFT #OTHEREFR ), and in those situations the optimization variable only includes Z.

The philosophy of the t-SVD model (1) is to minimize the rank in the transformed domain, which can model the implicit low-rankness of HSI.

There are naturally two key building blocks for exactly modeling the implicit low-rankness, i.e., the selection of the transform φ θ (•) and the exact low-rank characterization ψ(•) of the transformed frontal slice Z (k) .

Most t-SVD-based methods focus on the design of different transforms φ θ (•) (see examples in #OTHEREFR , #OTHEREFR , #OTHEREFR ), but all of them pay less attention to the exact characterization of the transformed frontal slice.

H2Tf For Hyperspectral Image Denoising: Where Hierarchical Nonlinear Transform Meets Hierarchical Matrix Factorization

Abstract: Recently, tensor singular value decomposition (t-SVD) has emerged as a promising tool for hyperspectral image (HSI) processing. In the t-SVD, there are two key building blocks: (i) the low-rank enhanced transform and (ii) the accompanying low-rank characterization of transformed frontal slices. Previous t-SVD methods mainly focus on the developments of (i), while neglecting the other important aspect, i.e., the exact characterization of transformed frontal slices. In this letter, we exploit the potentiality in both building blocks by leveraging the Hierarchical nonlinear transform and the Hierarchical matrix factorization to establish a new Tensor Factorization (termed as H2TF). Compared to shallow counter partners, e.g., low-rank matrix factorization or its convex surrogates, H2TF can better capture complex structures of transformed frontal slices due to its hierarchical modeling abilities. We then suggest the H2TF-based HSI denoising model and develop an alternating direction method of multipliers-based algorithm to address the resultant model. Extensive experiments validate the superiority of our method over state-of-the-art HSI denoising methods.

Framelet Representation Of Tensor Nuclear Norm For Third-Order Tensor Completion

Abstract: The main aim of this paper is to develop a framelet representation of the tensor nuclear norm for third-order tensor completion. In the literature, the tensor nuclear norm can be computed by using tensor singular value decomposition based on the discrete Fourier transform matrix, and tensor completion can be performed by the minimization of the tensor nuclear norm which is the relaxation of the sum of matrix ranks from all Fourier transformed matrix frontal slices. These Fourier transformed matrix frontal slices are obtained by applying the discrete Fourier transform on the tubes of the original tensor. In this paper, we propose to employ the framelet representation of each tube so that a framelet transformed tensor can be constructed. Because of framelet basis redundancy, the representation of each tube is sparsely represented. When the matrix slices of the original tensor are highly correlated, we expect the corresponding sum of matrix ranks from all framelet transformed matrix frontal slices would be small, and the resulting tensor completion can be performed much better. The proposed minimization model is convex and global minimizers can be obtained. Numerical results on several types of multi-dimensional data (videos, multispectral images, and magnetic resonance imaging data) have tested and shown that the proposed method outperformed the other testing tensor completion methods.


Finite Presentability Of Minor-Excluded Groups. A Walk In A Graph G Is A Finite Sequence Of Vertices

The following result was proved in #REFR Theorem 25] when G is a quasitransitive locally finite planar graph.

, v ′ ℓ ) are two walks such that v k = v ′ 1 , then their sum is the walk W • W ′ := (v 1 , . . . , v k = v ′ 1 , . . . , v ′ ℓ )

We will say that a set of closed walks W generates another set of closed walks W ′ if every element of W ′ can be obtained from elements of W by adding and deleting spurs and repetitions, and performing sums, reflections and rotations.

We reuse some of the arguments of the proof of #OTHEREFR Proposition 22] and combine them with our structure theorem to extend the result to graphs excluding K ∞ as a minor.

Let G be a locally finite graph excluding K ∞ as a minor and let Γ be a group acting quasi-transitively on G.

Then the set of closed walks of G admits a Γ-invariant generating set with finitely many Γ-orbits.

The Structure Of Quasi-Transitive Graphs Avoiding A Minor With Applications To The Domino Problem

Abstract: An infinite graph is quasi-transitive if its vertex set has finitely many orbits under the action of its automorphism group. In this paper we obtain a structure theorem for locally finite quasi-transitive graphs avoiding a minor, which is reminiscent of the Robertson-Seymour Graph Minor Structure Theorem. We prove that every locally finite quasi-transitive graph G avoiding a minor has a treedecomposition whose torsos are finite or planar; moreover the tree-decomposition is canonical, i.e. invariant under the action of the automorphism group of G. As applications of this result, we prove the following. • Every locally finite quasi-transitive graph attains its Hadwiger number, that is, if such a graph contains arbitrarily large clique minors, then it contains an infinite clique minor. This extends a result of Thomassen (1992) who proved it in the 4-connected case and suggested that this assumption could be omitted. In particular, this shows that a Cayley graph excludes a finite minor if and only if it avoids the countable clique as a minor. • Locally finite quasi-transitive graphs avoiding a minor are accessible (in the sense of Thomassen and Woess), which extends known results on planar graphs to any proper minor-closed family. • Minor-excluded finitely generated groups are accessible (in the group-theoretic sense) and finitely presented, which extends classical results on planar groups. • The domino problem is decidable in a minor-excluded finitely generated group if and only if the group is virtually free, which proves the minorexcluded case of a conjecture of Ballier and Stein (2018).

Planar Transitive Graphs

Abstract: We prove that the first homology group of every planar locally finite transitive graph $G$ is finitely generated as an $\Aut(G)$-module and we prove a similar result for the fundamental group of locally finite planar Cayley graphs. Corollaries of these results include Droms's theorem that planar groups are finitely presented and Dunwoody's theorem that planar locally finite transitive graphs are accessible.


As a consequence, researchers are forced to leverage synthetic data generated from images (e.g., #REFR ) for transformation learning in deep planar tracking, which may result in inferior performance Figure 2 .

In particular, several benchmarks (e.g., #OTHEREFR ) have been specially developed for evaluating and comparing different planar trackers, which greatly facilitates related research and progress on this topic.

Despite this, these benchmarks are severely limited in further pushing the frontier of planar object tracking.

One of the major issues with existing benchmarks is their relatively small scales.

Especially, in the deep learning era, to unleash the potential of deep planar tracking, it is desired to have a large-scale platform. Nevertheless, as displayed in Fig.

2 , currently all planar tracking benchmarks consist of less than 300 sequences, which is insufficient for largescale learning of deep planar tracking.

Summary of planar object tracking datasets, containing POT-280 #OTHEREFR , POT-210 #OTHEREFR , TMT #OTHEREFR , UCSB #OTHEREFR , Metiao #OTHEREFR , POIC #OTHEREFR , and PlanarTrack.

The circle diameter is in proportion to the number of frames of a dataset. Our PlanarTrack is the largest among all these benchmarks. due to domain gap between different tasks.

Besides the small-scale issue, another problem is the less challenging scenarios for planar object tracking.

Early planar tracking datasets (e.g., #OTHEREFR ) are constructed from the indoor laboratories with simple background, which cannot reflect the diverse and complicated scenarios of real world in performance evaluation.

To deal with this, recent datasets (e.g., #OTHEREFR ) directly collect videos in the wild.

Planartrack: A Large-Scale Challenging Benchmark For Planar Object Tracking

Abstract: Planar object tracking is a critical computer vision problem and has drawn increasing interest owing to its key roles in robotics, augmented reality, etc. Despite rapid progress, its further development, especially in the deep learning era, is largely hindered due to the lack of large-scale challenging benchmarks. Addressing this, we introduce PlanarTrack, a large-scale challenging planar tracking benchmark. Specifically, PlanarTrack consists of 1,000 videos with more than 490K images. All these videos are collected in complex unconstrained scenarios from the wild, which makes Planar-Track, compared with existing benchmarks, more challenging but realistic for real-world applications. To ensure the high-quality annotation, each frame in PlanarTrack is manually labeled using four corners with multiple-round careful inspection and refinement. To our best knowledge, Planar-Track, to date, is the largest and most challenging dataset dedicated to planar object tracking. In order to analyze the proposed PlanarTrack, we evaluate 10 planar trackers and conduct comprehensive comparisons and in-depth analysis. Our results, not surprisingly, demonstrate that current topperforming planar trackers degenerate significantly on the challenging PlanarTrack and more efforts are needed to improve planar tracking in the future. In addition, we further derive a variant named PlanarTrack BB for generic object tracking from PlanarTrack. Our evaluation of 10 excellent generic trackers on PlanarTrack BB manifests that, surprisingly, PlanarTrack BB is even more challenging than several popular generic tracking benchmarks and more attention should be paid to handle such planar objects, though they are rigid. All benchmarks and evaluations will be released at the project webpage. * Equal contributions. † Corresponding author. (a) Example of generic object tracking with rectangular bounding box

Microsoft Coco: Common Objects In Context

Abstract: We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.


Similar approaches have been presented for other problems, see e.g., Bamberger et al. #REFR for results on ∆ + 1 coloring.

(1) Better randomized constructions for network decomposition have been known.

In particular, the work of Linial and Saks #OTHEREFR presented an O(log 2 n) round randomized algorithm for computing decompositions with O(log n) colors and O(log n) weak diameter.

Elkin and Neiman #OTHEREFR imported a parallel algorithm of Miller, Peng and Xu #OTHEREFR into the distributed setting and obtained an O(log 2 n) round randomized algorithm for computing decompositions with O(log n) colors and O(log n) strong diameter.

(2) The deterministic MIS method described in Section 1.1 for using network decompositions in computing symmetry-breaking problems such as MIS would require large messages, as it gathers the topology of each cluster in a center.

For MIS, one can work with O(log n)-bit messages, by using a derandomization method of Censor-Hillel, Parter, and Schwartzman #OTHEREFR , and that gives a deterministic algorithm with O(log n)-bit messages and round complexity O(cd)•poly(log n).

(3) Ghaffari, Harris, and Kuhn #OTHEREFR and Ghaffari, Kuhn and Maus #OTHEREFR showed that one can get a general derandomization method for the LOCAL model using network decompositions.

This method transforms any poly(log n)-round randomized algorithm for any problem whose solution can be checked in poly(log n) rounds into a deterministic algorithm with round complexity O(cd + t) • poly(log n), assuming we have a deterministic (c, d) network decomposition algorithm with round complexity t.

Faster Deterministic Distributed Mis And Approximate Matching

Abstract: We present an O(log 2 n) round deterministic distributed algorithm for the maximal independent set problem. By known reductions, this round complexity extends also to maximal matching, ∆ + 1 vertex coloring, and 2∆ − 1 edge coloring. These four problems are among the most central problems in distributed graph algorithms and have been studied extensively for the past four decades. This improved round complexity comes closer to the Ω(log n) lower bound of maximal independent set and maximal matching [Balliu et al. FOCS '19]. The previous best known deterministic complexity for all of these problems was Θ(log 3 n). Via the shattering technique, the improvement permeates also to the corresponding randomized complexities, e.g., the new randomized complexity of ∆ + 1 vertex coloring is now O(log 2 log n) rounds. Our approach is a novel combination of the previously known (and seemingly orthogonal) two methods for developing fast deterministic algorithms for these problems, namely global derandomization via network decomposition (see e.g., [Rozhon, Ghaffari STOC'20; Ghaffari, Grunau, Rozhon SODA'21; Ghaffari et al. SODA'23]) and local rounding of fractional solutions (see e.g., [Fischer DISC'17; Harris FOCS'19; Fischer, Ghaffari, Kuhn FOCS'17; Ghaffari, Kuhn FOCS'21; Faour et al. SODA'23]). We consider a relaxation of the classic network decomposition concept, where instead of requiring the clusters in the same block to be non-adjacent, we allow each node to have a small number of neighboring clusters. We also show a deterministic algorithm that computes this relaxed decomposition faster than standard decompositions. We then use this relaxed decomposition to significantly improve the integrality of certain fractional solutions, before handing them to the local rounding procedure that now has to do fewer rounding steps. Randomized algorithms, and the pursuit of deterministic algorithms. In the 1980s, Luby [Lub86] and Alon, Babai, and Itai [ABI86] presented a simple and elegant randomized distributed algorithm that computes an MIS in O(log n) rounds, with high probability 1. Due to known reductions, this MIS algorithm led to O(log n) round randomized algorithms for many other key graph problems, including maximal matching, ∆ + 1 vertex coloring, and (2∆ − 1) edge coloring. These problems are often listed as the four fundamental symmetry-breaking problems in distributed graph algorithms and have a wide range of applications. The O(log n)-round randomized algorithm naturally led the researchers to seek a deterministic distributed algorithm with the same round complexity. In his celebrated work [Lin87, Lin92], Linial asked "can it [MIS] always be found in polylogarithmic time [deterministically]? " He even added that "getting a deterministic polylog-time algorithm for MIS seems hard." Since then, this became known as Linial's MIS question and turned into one of the research foci in distributed graph algorithms.

Efficient Deterministic Distributed Coloring With Small Bandwidth

Abstract: We show that the $(degree+1)$-list coloring problem can be solved deterministically in $O(D \cdot \log n \cdot\log^2\Delta)$ rounds in the \CONGEST model, where $D$ is the diameter of the graph, $n$ the number of nodes, and $\Delta$ the maximum degree. Using the recent polylogarithmic-time deterministic network decomposition algorithm by Rozho\v{n} and Ghaffari [STOC 2020], this implies the first efficient (i.e., $\poly\log n$-time) deterministic \CONGEST algorithm for the $(\Delta+1)$-coloring and the $(\mathit{degree}+1)$-list coloring problem. Previously the best known algorithm required $2^{O(\sqrt{\log n})}$ rounds and was not based on network decompositions. Our techniques also lead to deterministic $(\mathit{degree}+1)$-list coloring algorithms for the congested clique and the massively parallel computation (MPC) model. For the congested clique, we obtain an algorithm with time complexity $O(\log\Delta\cdot\log\log\Delta)$, for the MPC model, we obtain algorithms with round complexity $O(\log^2\Delta)$ for the linear-memory regime and $O(\log^2\Delta + \log n)$ for the sublinear memory regime.


In light of our observations that utilizing pretrained weights for both student and teacher networks leads to a improved performance, we note that the Noisy Student #REFR reported no improvement in performance when initializing both models simultaneously.

They claim that this approach may sometimes lead to getting stuck in a local optima, resulting in inferior performance compared to training the student model from scratch.

We hypothesize that this contrasting result could be due to the fact that they used the same pretrained weight for both teacher and student.

Since the student model has already inherited all the knowledge that could be learned from the teacher, distilling knowledge with the teacher may not bring additional performance gains.

A recent study by Allen-Zhu and Li #OTHEREFR revealed that different networks trained with distinct random seeds learn different knowledge.

Motivated by this observation, we train VGG-16 on CIFAR-100 with 8 different seeds following the same procedure as the baseline in Section 4.1 and utilize them to initialize teacher and student networks.

Bayesian Optimization Meets Self-Distillation

Abstract: Bayesian optimization (BO) has contributed greatly to improving model performance by suggesting promising hyperparameter configurations iteratively based on observations from multiple training trials. However, only partial knowledge (i.e., the measured performances of trained models and their hyperparameter configurations) from previous trials is transferred. On the other hand, Self-Distillation (SD) only transfers partial knowledge learned by the task model itself. To fully leverage the various knowledge gained from all training trials, we propose the BOSS framework, which combines BO and SD. BOSS suggests promising hyperparameter configurations through BO and carefully selects pre-trained models from previous trials for SD, which are otherwise abandoned in the conventional BO process. BOSS achieves significantly better performance than both BO and SD in a wide range of tasks including general image classification, learning with noisy labels, semi-supervised learning, and medical image analysis tasks.

Self-Training With Noisy Student Improves Imagenet Classification

Abstract: We present a simple self-training method that achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. On robustness test sets, it improves ImageNet-A top-1 accuracy from 61.0% to 83.7%, reduces ImageNet-C mean corruption error from 45.7 to 28.3, and reduces ImageNet-P mean flip rate from 27.8 to 12.2. ::: To achieve this result, we first train an EfficientNet model on labeled ImageNet images and use it as a teacher to generate pseudo labels on 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images. We iterate this process by putting back the student as the teacher. During the generation of the pseudo labels, the teacher is not noised so that the pseudo labels are as accurate as possible. However, during the learning of the student, we inject noise such as dropout, stochastic depth and data augmentation via RandAugment to the student so that the student generalizes better than the teacher.


For pMRI: Variational network (VN) #REFR introduced gradient descent method by applying given sensitivities {s i }.

Most existing deep-learning (DL) based methods rendering end-to-end neural networks mapping from the partial k-space data to the reconstructed images #OTHEREFR .

The common issue with this class of methods is that the DNNs require excessive amount of data to train, and the resulting networks perform similar to "black-boxes" which are difficult to interpret and modify.

In recent years, a class of DL based methods improve over the end-to-end training by selecting the scheme of an iterative optimization algorithm and prescribe a phase number T , map each iteration of the scheme to one phase of the network.

These methods are often known as the learned optimization algorithms (LOAs), #OTHEREFR .

For instance, ADMM-Net #OTHEREFR , ISTA-Net + #OTHEREFR , and cascade network #OTHEREFR are regular MRI reconstruction.

MoDL #OTHEREFR proposed a recursive network by unrolling the conjugate gradient algorithm using a weight sharing strategy.

Blind-PMRI-Net #OTHEREFR designed three network blocks to alternately update multi-channel images, sensitivity maps and the reconstructed MR image using an iterative algorithm based on half-quadratic splitting.

The network in #OTHEREFR developed a Bayesian framework for joint MRI-PET reconstruction. VS-Net #OTHEREFR derived a variable splitting optimization method.

However, existing methods still face the lack of accurate coil sensitivity maps and proper regularization in the pMRI problem.

Recently, a method called DeepcomplexMRI #OTHEREFR developed an end-to-end learning without explicitly using coil sensitivity maps to recover channel-wise images, and then combine to a single channel image in testing.

Optimization-Based Deep Learning Methods For Magnetic Resonance Imaging Reconstruction And Synthesis

Learning A Variational Network For Reconstruction Of Accelerated Mri Data

Abstract: Purpose: To allow fast and high-quality reconstruction of clinical accelerated multi-coil MR data by learning a variational network that combines the mathematical structure of variational models with deep learning. Theory and Methods: Generalized compressed sensing reconstruction formulated as a variational model is embedded in an unrolled gradient descent scheme. All parameters of this formulation, including the prior model defined by filter kernels and activation functions as well as the data term weights, are learned during an offline training procedure. The learned model can then be applied online to previously unseen data. Results: The variational network approach is evaluated on a clinical knee imaging protocol. The variational network reconstructions outperform standard reconstruction algorithms in terms of image quality and residual artifacts for all tested acceleration factors and sampling patterns. Conclusion: Variational network reconstructions preserve the natural appearance of MR images as well as pathologies that were not included in the training data set. Due to its high computational performance, i.e., reconstruction time of 193 ms on a single graphics card, and the omission of parameter tuning once the network is trained, this new approach to image reconstruction can easily be integrated into clinical workflow.