This finding weakens the claim made in #REFR of achieving a successful sampling task, namely achieving approximate sampling for some specific distribution, and may shed doubt on the potential of NISQ systems to achieve sampling tasks.

In this paper we studied three main issues, mostly from a statistical point of view.

The first issue, already demonstrated in #OTHEREFR is the large gap between the Google samples and the Google noise model or any specific noise model we studied.

The second issue concerns Formula (77) of the Google paper (4) with its simple independence-like form.

This, and the noise and calibration involved in the experimental process, make it surprising that Google's fidelity estimates appear to be rather close to Formula (77) for hundreds of different experiments.

Surprises do occur in science, and may lead to important scientific discoveries, but from a statistical point of view, the subsumed independence between components of systems such as quantum computers, which are known to be sensitive to noise and errors caused by interactions with their environment, is striking.

Related questions arise from the systematic deviation of Formula (77) for patch circuits, and the large difference between the F XEB fidelity of the two patches. The remarkable verification of Kalachev et al. #OTHEREFR and Liu et al.

#OTHEREFR weakened these concerns by showing that Formula (77) provides a good approximation to hundreds of circuits for which the Google team stated that they had not computed the amplitudes.

Questions And Concerns About Google'S Quantum Supremacy Claim

Abstract: In October 2019, Nature published a paper [6] describing an experimental work that was performed at Google. The paper claims to demonstrate quantum (computational) supremacy on a 53-qubit quantum computer. Since then we have been involved in a long-term project to study various statistical aspects of the Google experiment. In [30] we studied Google's statistical framework that we found to be very sound and offered some technical improvements. This document describes three main concerns (based on statistical analysis) about the Google 2019 experiment. The first concern is that the data do not agree with Google's noise model (or any other specific model). The second concern is that a crucial simple formula for a priori estimation of the fidelity seems to involve an unexpected independence assumption, and yet it gives very accurate predictions. The third concern is about statistical properties of the calibration process. 1

Supplementary Information For"Quantum Supremacy Using A Programmable Superconducting Processor"

Abstract: The promise of quantum computers is that certain computational tasks might be executed exponentially faster on a quantum processor than on a classical processor1. A fundamental challenge is to build a high-fidelity processor capable of running quantum algorithms in an exponentially large computational space. Here we report the use of a processor with programmable superconducting qubits2–7 to create quantum states on 53 qubits, corresponding to a computational state-space of dimension 253 (about 1016). Measurements from repeated experiments sample the resulting probability distribution, which we verify using classical simulations. Our Sycamore processor takes about 200 seconds to sample one instance of a quantum circuit a million times—our benchmarks currently indicate that the equivalent task for a state-of-the-art classical supercomputer would take approximately 10,000 years. This dramatic increase in speed compared to all known classical algorithms is an experimental realization of quantum supremacy8–14 for this specific computational task, heralding a much-anticipated computing paradigm. Quantum supremacy is demonstrated using a programmable superconducting processor known as Sycamore, taking approximately 200 seconds to sample one instance of a quantum circuit a million times, which would take a state-of-the-art supercomputer around ten thousand years to compute.


Alternatively, the transmitter can predict its future haptic data and transmit the predicted data to compensate for the transmission delay #REFR .

In the tele-operated needle insertion, the force/torque feedback from the patient is predicted by inputting the force/torque commands of the surgeon to the hidden Markov model (HMM) #OTHEREFR .

Audiovisual data collected in the interaction with a surface material are input to a neural network-based semantic learning algorithm to predict the texture of the surface material #OTHEREFR .

Haptic data can be predicted either at the receiver side or at the transmitter side to compensate for an excessive delay or packet loss.

The receiver can predict the haptic data from the transmitter when an excessive delay occurs #OTHEREFR .

For example, digital twin-based prediction can be used by the receiver for low-latency interactions #OTHEREFR .

In this case, the prediction of whether haptic interaction is about to occur can assist to determine whether the haptic data prediction and the subsequent transmission are necessary #OTHEREFR .

Haptic data prediction algorithms, such as AI-based ones, can be computing-intensive.

To this end, they can be implemented using computing resources in the network to satisfy the stringent delay requirements #OTHEREFR , #OTHEREFR .

In a teleoperation scenario, each of the two interacting haptic devices is associated with one edge server which caches the haptic interaction data, trains and implements the LSTM networkbased prediction algorithm, and delivers the predicted haptic data to its associated haptic device #OTHEREFR .

Furthermore, with close proximity, auxiliary robots can be deployed around haptic devices to implement haptic data prediction and deliver the results to the devices using device-to-device (D2D) communications #OTHEREFR .

Toward Immersive Communications In 6G

Abstract: The sixth generation (6G) networks are expected to enable immersive communications and bridge the physical and the virtual worlds. Integrating extended reality, holography, and haptics, immersive communications will revolutionize how people work, entertain, and communicate by enabling lifelike interactions. However, the unprecedented demand for data transmission rate and the stringent requirements on latency and reliability create challenges for 6G networks to support immersive communications. In this survey article, we present the prospect of immersive communications and investigate emerging solutions to the corresponding challenges for 6G. First, we introduce use cases of immersive communications, in the fields of entertainment, education, and healthcare. Second, we present the concepts of immersive communications, including extended reality, haptic communication, and holographic communication, their basic implementation procedures, and their requirements on networks in terms of transmission rate, latency, and reliability. Third, we summarize the potential solutions to addressing the challenges from the aspects of communication, computing, and networking. Finally, we discuss future research directions and conclude this study.

Prediction And Communication Co-Design For Ultra-Reliable And Low-Latency Communications

Abstract: Ultra-reliable and low-latency communications (URLLC) are considered as one of three new application scenarios in the fifth generation cellular networks. In this work, we aim to reduce the user experienced delay through prediction and communication co-design, where each mobile device predicts its future states and sends them to a data center in advance. Since predictions are not error-free, we consider prediction errors and packet losses in communications when evaluating the reliability of the system. Then, we formulate an optimization problem that maximizes the number of URLLC services supported by the system by optimizing time and frequency resources and the prediction horizon. Simulation results verify the effectiveness of the proposed method, and show that the tradeoff between user experienced delay and reliability can be improved significantly via prediction and communication co-design. Furthermore, we carried out an experiment on the remote control in a virtual factory, and validated our concept on prediction and communication co-design with the practical mobility data generated by a real tactile device.


For the graph decoder, we use the 6-layer transformer encoder with the pre-layer normalization #REFR and 16 heads.

We reuse the prototype decoder for the parameter estimation; we add another projection head, append a task token xT to differentiate the two tasks, and remove the causal attention mask.

Since each parameter has a different range and scale, we translate and rescale the ground-truth value to fit into [0, 1] range. Architecture Details.

The FFT size, hop length, and the number of Mel filter banks of the reference Mel spectrogram are 1536, 384, and 256, respectively.

The convolutional backbone is a VGGish model #OTHEREFR with the following modifications: (i) depthwise separable convolutions #OTHEREFR , (ii) channels divided into four groups with dilations of #OTHEREFR , (iii) and the use of layer normalization #OTHEREFR .

We used a transformer decoder layer with 1 (singing) and 6 (drum) queries for the pooling.

While the original paper used eigenvectors of the normalized Laplacian matrix as node id encoding, we use the sinusoidal embeddings since the eigenvectors are intractable during the decoding. Training.

We train the prototype decoding task in a teacher-forcing manner using the cross-entropy losses with label-smoothing of 0.1.

At the same time, we train the parameter estimation task by feeding an oracle prototype G0 as input and using the l1 distance as an objective.

We use AdamW #OTHEREFR optimizer, a linear learning rate scheduler with 5e-4 peak learning rate, 50k warmup steps, 200k total training steps, and batch size of 32.

Blind Estimation Of Audio Processing Graph

Abstract: Musicians and audio engineers sculpt and transform their sounds by connecting multiple processors, forming an audio processing graph. However, most deep-learning methods overlook this real-world practice and assume fixed graph settings. To bridge this gap, we develop a system that reconstructs the entire graph from a given reference audio. We first generate a realistic graph-reference pair dataset and train a simple blind estimation system composed of a convolutional reference encoder and a transformer-based graph decoder. We apply our model to singing voice effects and drum mixing estimation tasks. Evaluation results show that our method can reconstruct complex signal routings, including multi-band processing and sidechaining.

On Layer Normalization In The Transformer Architecture

Abstract: The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyper-parameter tunings. In this paper, we first study theoretically why the learning rate warm-up stage is essential and show that the location of layer normalization matters. Specifically, we prove with mean field theory that at initialization, for the original-designed Post-LN Transformer, which places the layer normalization between the residual blocks, the expected gradients of the parameters near the output layer are large. Therefore, using a large learning rate on those gradients makes the training unstable. The warm-up stage is practically helpful for avoiding this problem. On the other hand, our theory also shows that if the layer normalization is put inside the residual blocks (recently proposed as Pre-LN Transformer), the gradients are well-behaved at initialization. This motivates us to remove the warm-up stage for the training of Pre-LN Transformers. We show in our experiments that Pre-LN Transformers without the warm-up stage can reach comparable results with baselines while requiring significantly less training time and hyper-parameter tuning on a wide range of applications.


Inspired by K-FAC, there have been other works discussing approximations of G θ and its inverse #REFR .

where λ > 0 is the so-called Tikhonov regularization parameter.

It is well-known #OTHEREFR , #OTHEREFR , that under the assumption of approximating the model f with its first-order Taylor expansion, the Hessian corresponds with the so-called generalized Gauss-Newton (GGN) matrix G θ , and hence (4) can be expressed as

A major practical limitation of (5) is the computation of the inverse term.

A method that alleviates this difficulty is known as Kronecker-Factored Approximate Curvature (K-FAC) #OTHEREFR which approximates the block-diagonal (i.e., layer-wise) empirical Hessian or GGN matrix.

In the following, we discuss a popular approach that allows for (moderately) efficient computation.

The generalized Gauss-Newton matrix G θ is defined as

where J and ∇ 2 denote the Jacobian and Hessian matrices, respectively.

Correspondingly, the diagonal block of G θ corresponding to the weights of the ith layer

Isaac Newton: Input-Based Approximate Curvature For Newton'S Method

Abstract: We present ISAAC (Input-baSed ApproximAte Curvature), a novel method that conditions the gradient using selected second-order information and has an asymptotically vanishing computational overhead, assuming a batch size smaller than the number of neurons. We show that it is possible to compute a good conditioner based on only the input to a respective layer without a substantial computational overhead. The proposed method allows effective training even in small-batch stochastic regimes, which makes it competitive to first-order as well as second-order methods.

Backpack: Packing More Into Backprop

Abstract: Automatic differentiation frameworks are optimized for exactly one thing: computing the average mini-batch gradient. Yet, other quantities such as the variance of the mini-batch gradients or many approximations to the Hessian can, in theory, be computed efficiently, and at the same time as the gradient. While these quantities are of great interest to researchers and practitioners, current deep learning software does not support their automatic calculation. Manually implementing them is burdensome, inefficient if done naively, and the resulting code is rarely shared. This hampers progress in deep learning, and unnecessarily narrows research to focus on gradient descent and its variants; it also complicates replication studies and comparisons between newly developed methods that require those quantities, to the point of impossibility. To address this problem, we introduce BackPACK, an efficient framework built on top of PyTorch, that extends the backpropagation algorithm to extract additional information from first-and second-order derivatives. Its capabilities are illustrated by benchmark reports for computing additional quantities on deep neural networks, and an example application by testing several recent curvature approximations for optimization.


Meta-SGD #REFR is a well-known constant diagonal preconditioner (i.e., diag(a 1 , • • • , a n )) that does not need to be positive definite and we also investigate its modification with a constraint on positive definiteness. The results are shown in Table 8 .

simple constant preconditioners: Approximate GAP is a low-complexity method where SVD Meta-SGD #OTHEREFR 2.4218 × 10 5 100.0% MC #OTHEREFR 2.7106 × 10 6 2140.4% PAMELA #OTHEREFR 1.6239 × 10 5 34.1% MH #OTHEREFR 7.2196 × 10 7 59586.7% Sparse-MAML #OTHEREFR 2.4218 × 10 5 100.0% GAP 1.2131 × 10 5 0.2% Table 9 . Ablation study of PGAP on mini-ImageNet.

Performance of the GAP-trained model is significantly affected by not applying PGAP.

operation is avoided by approximating GAP with a constant diagonal preconditioner.

A natural question to ask is how does Approximate GAP compare with other constant diagonal preconditioners.

To answer this question, we have compared Approximate GAP with Meta-SGD and a modified Meta-SGD.

It can be observed that enforcing positive definiteness can improve Meta-SGD.

Furthermore, an additional improvement can be achieved by Approximate GAP.

While both modified Meta-SGD and Approximate GAP are positive definite, Approximate GAP is different because it inherits an additional constraint from GAP -a block diagonal structure where a constant diagonal matrix M is repeated (i.e., blkdiag(M, • • • , M )).

The inherited constraint provides a gain over the modified Meta-SGD.

Does GAP learn a useful preconditioner: While a Riemannian metric can be helpful, it does not mean any Riemannian metric will result in an improvement.

Meta-Learning With A Geometry-Adaptive Preconditioner

Abstract: Model-agnostic meta-learning (MAML) is one of the most successful meta-learning algorithms. It has a bi-level optimization structure where the outer-loop process learns a shared initialization and the inner-loop process optimizes task-specific weights. Although MAML relies on the standard gradient descent in the inner-loop, recent studies have shown that controlling the inner-loop's gradient descent with a meta-learned preconditioner can be beneficial. Existing preconditioners, however, cannot simultaneously adapt in a task-specific and path-dependent way. Additionally, they do not satisfy the Riemannian metric condition, which can enable the steepest descent learning with preconditioned gradient. In this study, we propose Geometry-Adaptive Preconditioned gradient descent (GAP) that can overcome the limitations in MAML; GAP can efficiently meta-learn a preconditioner that is dependent on task-specific parameters, and its preconditioner can be shown to be a Riemannian metric. Thanks to the two properties, the geometryadaptive preconditioner is effective for improving the innerloop optimization. Experiment results show that GAP outperforms the state-of-the-art MAML family and preconditioned gradient descent-MAML (PGD-MAML) family in a variety of few-shot learning tasks. Code is available at: https://github.com/Suhyun777/CVPR23-GAP.

Meta-Sgd: Learning To Learn Quickly For Few-Shot Learning

Abstract: Few-shot learning is challenging for learning algorithms that learn each task in isolation and from scratch. In contrast, meta-learning learns from many related tasks a meta-learner that can learn a new task more accurately and faster with fewer examples, where the choice of meta-learners is crucial. In this paper, we develop Meta-SGD, an SGD-like, easily trainable meta-learner that can initialize and adapt any differentiable learner in just one step, on both supervised learning and reinforcement learning. Compared to the popular meta-learner LSTM, Meta-SGD is conceptually simpler, easier to implement, and can be learned more efficiently. Compared to the latest meta-learner MAML, Meta-SGD has a much higher capacity by learning to learn not just the learner initialization, but also the learner update direction and learning rate, all in a single meta-learning process. Meta-SGD shows highly competitive performance for few-shot learning on regression, classification, and reinforcement learning.


Nonetheless, this study is limited in its scope of long-form QA task on ELI5 #REFR dataset, and further research is required to explore more elaborate knowledge analysis for more tasks and to evaluate the generalizability of our findings.

This suggests that ChatGPT is becoming better at generating responses that contain semantic frames or frames used to represent entities and events.

Similarly, we can observe an decreasing trend in the frequency of argument role features such as Agent, Patient, and Theme in ChatGPT's responses over time.

This indicates an improvement in ChatGPT's ability to recognize the argument roles of entities and events in generating responses that convey the meaning correctly.

Overall, the results presented in Figure 6 , 7 and 8 demonstrate that ChatGPT's patterns on knowledge are showing improvement over time.

The decreasing frequency values of named entity, relation, opinion and frame features suggest that ChatGPT has become more proficient in generating responses that contain less complex knowledge to fit the instruction "Explain like I am five".

P e r s o n -0 1 -8 O r g a n iz a t io n -0 1 -1 8 L o c a t io n -0 1 -1 8 W o r k -0 1 -1 8 T im e -0 1 -1 R e la t io n -0 1 -1 8 P e r s o n -0 3 -0 5 O r g a n iz a t io n -0 3 -0 5 L o c a t io n -0 3 -0 5 W o r k -0 3 -0 5 T im e -0 3 -0 5 R e la t io n -0 3 -0 5 P e r s o n -0 4 -0 9 O r g a n iz a t io n -0 4 -0 9 L o c a t io n -0 4 -0 9 W o r k -0 4 -0 9 T im e -0 4 -0 9 R e la t io n -0 4 -0 9 O p in io n -0 1 -1 8 S e n t im e n t -0 1 -1 8 O p in io n -0 3 -0 5 S e n t im e n t -0 3 -0 5 O p in io n -0 4 -0 9 S e n t im e n t -0 4 -0 9

Chatlog: Recording And Analyzing Chatgpt Across Time

Abstract: While there are abundant researches about evaluating ChatGPT on natural language understanding and generation tasks, few studies have investigated how ChatGPT's behavior changes over time. In this paper, we collect a coarse-to-fine temporal dataset called ChatLog, consisting of two parts that update monthly and daily: ChatLog-Monthly is a dataset of 38,730 question-answer pairs collected every month including questions from both the reasoning and classification tasks. ChatLog-Daily, on the other hand, consists of ChatGPT's responses to 1000 identical questions for long-form generation every day. We conduct comprehensive automatic and human evaluation to provide the evidence for the existence of ChatGPT evolving patterns. We further analyze the unchanged characteristics of ChatGPT over time by extracting its knowledge and linguistic features. We find some stable features to improve the robustness of a RoBERTa-based detector on new versions of ChatGPT. We will continuously maintain our project at https://github.com/ THU-KEG/ChatLog.

Eli5: Long Form Question Answering

Abstract: We introduce the first large-scale corpus for long-form question answering, a task requiring elaborate and in-depth answers to open-ended questions. The dataset comprises 270K threads from the Reddit forum ``Explain Like I'm Five'' (ELI5) where an online community provides answers to questions which are comprehensible by five year olds. Compared to existing datasets, ELI5 comprises diverse questions requiring multi-sentence answers. We provide a large set of web documents to help answer the question. Automatic and human evaluations show that an abstractive model trained with a multi-task objective outperforms conventional Seq2Seq, language modeling, as well as a strong extractive baseline. However, our best model is still far from human performance since raters prefer gold responses in over 86% of cases, leaving ample opportunity for future improvement.


Following this framework, semiring semantics for full first-order logic (FO) were developed in #REFR .

Prior to this work, these adaptations of team semantics have been studied in isolation from one another.

Data provenance provides means to describe the origins of data, allowing to give information about the witnesses to a query, or determining how a certain output is derived.

Provenance semirings were introduced in #OTHEREFR to devise a general framework that allows to uniformly treat extensions of positive relational algebra, where the tuples have annotations that reflect very diverse information.

Some motivating examples of said relations come from incomplete and probabilistic databases, and bag semantics.

This semiring framework captures a notion of data provenance called howprovenance, where the semiring operations essentially capture how each output is produced from the source.

The semiring semantics for FO refines, in particular, the classical Boolean semantics by allowing formulae to be evaluated as values from a semiring.

This allows for example counting proof trees, or winning strategies in the model checking game for A and φ.

In databases, dependencies are applied as integrity constraints (ICs) that specify sets of rules that the database needs to satisfy.

Formal analysis of the rules is facilitated by viewing them as FO sentences that usually follow certain syntactic patterns.

This approach is sometimes inadequate because query languages such as SQL operate with multisets (i.e., bags) of tuples instead of sets. Recently, (Chu et al.

Unified Foundations Of Team Semantics Via Semirings

Abstract: Semiring semantics for first-order logic provides a way to trace how facts represented by a model are used to deduce satisfaction of a formula. Team semantics is a framework for studying logics of dependence and independence in diverse contexts such as databases, quantum mechanics, and statistics by extending first-order logic with atoms that describe dependencies between variables. Combining these two, we propose a unifying approach for analysing the concepts of dependence and independence via a novel semiring team semantics, which subsumes all the previously considered variants for first-order team semantics. In particular, we study the preservation of satisfaction of dependencies and formulae between different semirings. In addition we create links to reasoning tasks such as provenance, counting, and repairs.

Semiring Provenance For First-Order Model Checking

Abstract: Given a first-order sentence, a model-checking computation tests whether the sentence holds true in a given finite structure. Data provenance extracts from this computation an abstraction of the manner in which its result depends on the data items that describe the model. Previous work on provenance was, to a large extent, restricted to the negation-free fragment of first-order logic and showed how provenance abstractions can be usefully described as elements of commutative semirings --- most generally as multivariate polynomials with positive integer coefficients. ::: In this paper we introduce a novel approach to dealing with negation and a corresponding commutative semiring of polynomials with dual indeterminates. These polynomials are used to perform reverse provenance analysis, i.e., finding models that satisfy various properties under given provenance tracking assumptions.


These results show that the proposed model works well on both simple and complex tables and outperforms EDD #REFR ) by about 8.1% and the best model in (Nassar et al., 2022) by about 2%.

In this experiment, we evaluate the effectiveness of WSTabNet for recognizing the structure of the table images on PubTabNet and FinTabNet datasets. Table  5 shows the table structure recognition performance (TEDS-struc.

scores) of the proposed model and the previous table structure recognition methods on the validation set of the PubTabNet and the test set of the FinTabNet.

On the test set of the FinTabNet dataset, the proposed model achieved TEDS-struc.

of 99.06%, 98.33%, and 98.72% on the simple, complex, and all table images, respectively.

On the validation set of the PubTabNet dataset, the proposed model achieved TEDS-struc.

of 97.74% on all table images which again improves EDD by about 7.8% and the best model in (Nassar et al., 2022) by about 1%.

Note that all other methods except EDD are fully supervised approaches that require both HTML and cell bounding boxes annotations in the training step. (FT) Model was trained on PubTabNet and then finetuned.

Rethinking Image-Based Table Recognition Using Weakly Supervised Methods

Abstract: Most of the previous methods for table recognition rely on training datasets containing many richly annotated table images. Detailed table image annotation, e.g., cell or text bounding box annotation, however, is costly and often subjective. In this paper, we propose a weakly supervised model named WSTabNet for table recognition that relies only on HTML (or LaTeX) code-level annotations of table images. The proposed model consists of three main parts: an encoder for feature extraction, a structure decoder for generating table structure, and a cell decoder for predicting the content of each cell in the table. Our system is trained end-to-end by stochastic gradient descent algorithms, requiring only table images and their ground-truth HTML (or LaTeX) representations. To facilitate table recognition with deep learning, we create and release WikiTableSet, the largest publicly available image-based table recognition dataset built from Wikipedia. WikiTableSet contains nearly 4 million English table images, 590K Japanese table images, and 640k French table images with corresponding HTML representation and cell bounding boxes. The extensive experiments on WikiTableSet and two large-scale datasets: FinTabNet and PubTabNet demonstrate that the proposed weakly supervised model achieves better, or similar accuracies compared to the state-of-the-art models on all benchmark datasets.

Image-Based Table Recognition: Data, Model, And Evaluation

Abstract: Important information that relates to a specific topic in a document is often organized in tabular format to assist readers with information retrieval and comparison, which may be difficult to provide in natural language. However, tabular data in unstructured digital documents, e.g., Portable Document Format (PDF) and images, are difficult to parse into structured machine-readable format, due to complexity and diversity in their structure and style. To facilitate image-based table recognition with deep learning, we develop the largest publicly available table recognition dataset PubTabNet (this https URL), containing 568k table images with corresponding structured HTML representation. PubTabNet is automatically generated by matching the XML and PDF representations of the scientific articles in PubMed Central Open Access Subset (PMCOA). We also propose a novel attention-based encoder-dual-decoder (EDD) architecture that converts images of tables into HTML code. The model has a structure decoder which reconstructs the table structure and helps the cell decoder to recognize cell content. In addition, we propose a new Tree-Edit-Distance-based Similarity (TEDS) metric for table recognition. The experiments demonstrate that the EDD model can accurately recognize complex tables solely relying on the image representation, outperforming the state-of-the-art by 7.7% absolute TEDS score.


This is in contrast to previous approaches which were based on incorporating ab-initio orbitals #REFR , which could not reach chemical accuracy even for small molecules.

Therefore during supervised pre-training, the undetermined sign of the reference orbitals becomes irrelevant, leading to faster convergence as demonstrated in Sec. 2.4.

• Locality: When using localized HF-orbitals as input, the resulting TAOs are also localized.

Localized HF-orbitals are orbitals which have non-zero orbital features c Ik only on some subset of atoms.

Since we enforce the backflow f a θ to be antisymmetric (and thus f a (0) = 0), the resulting TAOs have zero contribution from atoms I with c Ik = 0.

• High expressivity: We empirically find that our ansatz is sufficiently expressive to model ground-state wavefunctions to high accuracy.

Towards A Foundation Model For Neural Network Wavefunctions

Abstract: Deep neural networks have become a highly accurate and powerful wavefunction ansatz in combination with variational Monte Carlo methods for solving the electronic Schrödinger equation. However, despite their success and favorable scaling, these methods are still computationally too costly for wide adoption. A significant obstacle is the requirement to optimize the wavefunction from scratch for each new system, thus requiring long optimization. In this work, we propose a novel neural network ansatz, which effectively maps uncorrelated, computationally cheap Hartree-Fock orbitals, to correlated, high-accuracy neural network orbitals. This ansatz is inherently capable of learning a single wavefunction across multiple compounds and geometries, as we demonstrate by successfully transferring a wavefunction model pre-trained on smaller fragments to larger compounds. Furthermore, we provide ample experimental evidence to support the idea that extensive pre-training of a such a generalized wavefunction model across different compounds and geometries could lead to a foundation wavefunction model. Such a model could yield high-accuracy ab-initio energies using only minimal computational effort for fine-tuning and evaluation of observables.

Deep Neural Network Solution Of The Electronic Schrödinger Equation.

Abstract: The electronic Schrodinger equation describes fundamental properties of molecules and materials, but can only be solved analytically for the hydrogen atom. The numerically exact full configuration-interaction method is exponentially expensive in the number of electrons. Quantum Monte Carlo is a possible way out: it scales well to large molecules, can be parallelized, and its accuracy has, as yet, only been limited by the flexibility of the used wave function ansatz. Here we propose PauliNet, a deep-learning wave function ansatz that achieves nearly exact solutions of the electronic Schrodinger equation. PauliNet has a multireference Hartree-Fock solution built in as a baseline, incorporates the physics of valid wave functions, and is trained using variational quantum Monte Carlo (VMC). PauliNet outperforms comparable state-of-the-art VMC ansatzes for atoms, diatomic molecules and a strongly-correlated hydrogen chain by a margin and is yet computationally efficient. We anticipate that thanks to the favourable scaling with system size, this method may become a new leading method for highly accurate electronic-strucutre calculations on medium-sized molecular systems.


Our code is implemented using PyTorch #REFR and PyTorch-Geometric , and all our experiments are ran on Nvidia RTX3090 GPUs with 24GB of memory.

RFP is compared with a number of popular and recent methods.

In particular, we focus on the comparison of RNF (Abboud et al., 2020) , Laplacian PE #OTHEREFR and PowerEmbed #OTHEREFR which is arguably the closest approach to RFP.

Additionally, we introduce further baselines based on our RFP method, such as including both the RFP and eigenvectors of the propagation operators without a complete RFP trajectory in order to demonstrate the importance of considering the complete trajectory.

For the baseline case of Laplacian eigenvectors only (denoted byL EIGVECS † ) we use the eigenvectors corresponding to the smallest eigenvalues, in order to be consistent with the literature #OTHEREFR .

Graph Positional Encoding Via Random Feature Propagation

Abstract: Two main families of node feature augmentation schemes have been explored for enhancing GNNs: random features and spectral positional encoding. Surprisingly, however, there is still no clear understanding of the relation between these two augmentation schemes. Here we propose a novel family of positional encoding schemes which draws a link between the above two approaches and improves over both. The new approach, named Random Feature Propagation (RFP), is inspired by the power iteration method and its generalizations. It concatenates several intermediate steps of an iterative algorithm for computing the dominant eigenvectors of a propagation matrix, starting from random node features. Notably, these propagation steps are based on graph-dependent propagation operators that can be either predefined or learned. We explore the theoretical and empirical benefits of RFP. First, we provide theoretical justifications for using random features, for incorporating early propagation steps, and for using multiple random initializations. Then, we empirically demonstrate that RFP significantly outperforms both spectral PE and random features in multiple node classification and graph classification benchmarks.

Pytorch: An Imperative Style, High-Performance Deep Learning Library

Abstract: Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it provides an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. ::: In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. ::: We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several common benchmarks.


At the same time, in near-stationary environments (P T is small enough), our result can recover to the performance of LogUCB1 algorithm #REFR .

For all γ ∈ (1/T, 1), λ = d log(T )/c µ , the dynamic regret of SCB-WeightUCB is bounded with probability at least 1 − 1/T , by

R T ≤ O k 2 µ √ c µ 1 (1 − γ) 3 /2 P T + k µ √ c µ d(1 − γ) 1 /2 T .

By setting γ = 1 − max{1/T, k µ P T /(dT )}, we achieve R T ≤          O k 5/4 µ c 1/2 µ d 3 /4 P 1 /4 T T 3 /4 when P T ≥ d kµT , O kµ c 1/2 µ d √ T when 0 ≤ P T < d kµT .

Compared to GLB, we improve the order of c µ from c −1 µ to c − 1 /2 µ by exploiting the self-concordant properties.

The proof of Theorem 3 is presented in Appendix C.2.

In addition, for the piecewise-stationary SCB, we propose SCB-PW-WeightUCB algorithm that gets rid of influence of c µ and thus directly improves upon .

For all γ ∈ (1/2, 1), D = log(T )/ log(1/γ) and λ = d log(T )/c µ , the regret of SCB-PW-WeightUCB is bounded with probability at least 1 − 1/T , by

R T ≤ O 1 1 − γ Γ T + 1 √ 1 − γ + d (1 − γ)T .

Revisiting Weighted Strategy For Non-Stationary Parametric Bandits

Abstract: Non-stationary parametric bandits have attracted much attention recently. There are three principled ways to deal with non-stationarity, including sliding-window, weighted, and restart strategies. As many non-stationary environments exhibit gradual drifting patterns, the weighted strategy is commonly adopted in real-world applications. However, previous theoretical studies show that its analysis is more involved and the algorithms are either computationally less efficient or statistically suboptimal. This paper revisits the weighted strategy for non-stationary parametric bandits. In linear bandits (LB), we discover that this undesirable feature is due to an inadequate regret analysis, which results in an overly complex algorithm design. We propose a refined analysis framework, which simplifies the derivation and importantly produces a simpler weight-based algorithm that is as efficient as window/restart-based algorithms while retaining the same regret as previous studies. Furthermore, our new framework can be used to improve regret bounds of other parametric bandits, including Generalized Linear Bandits (GLB) and Self-Concordant Bandits (SCB). For example, we develop a simple weighted GLB algorithm with an O(k 5 /4 µ c − 3 /4 µ d 3 /4 P 1 /4 T T 3 /4) regret, improving the O(k 2 µ c −1 µ d 9 /10 P 1 /5 T T 4 /5) bound in prior work, where k µ and c µ characterize the reward model's nonlinearity, P T measures the non-stationarity, d and T denote the dimension and time horizon.

Improved Optimistic Algorithms For Logistic Bandits

Abstract: The generalized linear bandit framework has attracted a lot of attention in recent years by extending the well-understood linear setting and allowing to model richer reward structures. It notably covers the logistic model, widely used when rewards are binary. For logistic bandits, the frequentist regret guarantees of existing algorithms are $\tilde{\mathcal{O}}(\kappa \sqrt{T})$, where $\kappa$ is a problem-dependent constant. Unfortunately, $\kappa$ can be arbitrarily large as it scales exponentially with the size of the decision set. This may lead to significantly loose regret bounds and poor empirical performance. In this work, we study the logistic bandit with a focus on the prohibitive dependencies introduced by $\kappa$. We propose a new optimistic algorithm based on a finer examination of the non-linearities of the reward function. We show that it enjoys a $\tilde{\mathcal{O}}(\sqrt{T})$ regret with no dependency in $\kappa$, but for a second order term. Our analysis is based on a new tail-inequality for self-normalized martingales, of independent interest.


The linear kernel SVC implementation LinearSVC from Scikit-learn library #REFR was used to conduct the experiment.

The experiments were repeated with the SVC model to the same two datasets.

Linear kernel SVC was chosen due to large dataset size.

Other kernel types were tested, however they did not scale to the large number of samples.

The samples were normalized using the Scikit-learn StandardScaler to ease the model convergence. Table 2 displays the results from the experiments.

The proposed method is clearly superior to the conventional method in both experiments.

The performance in the statistical change dataset is considerably worse when compared to the shift change dataset.

Improved Difference Images For Change Detection Classifiers In Sar Imagery Using Deep Learning

Abstract: Satellite-based Synthetic Aperture Radar (SAR) images can be used as a source of remote sensed imagery regardless of cloud cover and day-night cycle. However, the speckle noise and varying image acquisition conditions pose a challenge for change detection classifiers. This paper proposes a new method of improving SAR image processing to produce higher quality difference images for the classification algorithms. The method is built on a neural network-based mapping transformation function that produces artificial SAR images from a location in the requested acquisition conditions. The inputs for the model are: previous SAR images from the location, imaging angle information from the SAR images, digital elevation model, and weather conditions. The method was tested with data from a location in NorthEast Finland by using Sentinel-1 SAR images from European Space Agency, weather data from Finnish Meteorological Institute, and a digital elevation model from National Land Survey of Finland. In order to verify the method, changes to the SAR images were simulated, and the performance of the proposed method was measured using experimentation where it gave substantial improvements to performance when compared to a more conventional method of creating difference images.

Scikit-Learn: Machine Learning In Python

Abstract: Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing machine learning to non-specialists using a general-purpose high-level language. Emphasis is put on ease of use, performance, documentation, and API consistency. It has minimal dependencies and is distributed under the simplified BSD license, encouraging its use in both academic and commercial settings. Source code, binaries, and documentation can be downloaded from http://scikit-learn.sourceforge.net.


HMMR #REFR learns the human dynamics to predict pose and shape for past and future frames.

Regression-based scheme has recently received extensive research #OTHEREFR Lin et al., 2021b,a) , due to its directness and effectiveness.

HMR #OTHEREFR , is the representative regression-based methods, using an image encoder and regressor to predict the pose, shape and camera parameters.

To train the model well and make sure the realistic of the pose and shape, the reprojection loss and adversarial loss are introduced to leverage unpaired 2D-to-3D supervision.

In addition, several non-parameteric mesh regression methods are proposed to directly regress the mesh vertices coordinates, including Pose2Mesh #OTHEREFR , Convolution Mesh Regression #OTHEREFR , I2L-MeshNet , and the Transformer-based METRO #OTHEREFR and Mesh Graphormer #OTHEREFR .

Beyond estimating pose and shape from a singe image, video-based methods consider to fully dig the temporal motion information hidden in video data to improve the accuracy and robustness.

VIBE #OTHEREFR encodes temporal feature using a GRU and adopts an adversarial learning framework to learn kinematically plausible motion from a large-scale motion capture dataset.

TCMR #OTHEREFR introduces PoseForecast to forecast additional temporal features from past and future frames without a current frame.

MAED #OTHEREFR proposes to use a spatial-temporal encoder to learn temporally enhanced image features and regress the joint rotations following a defined kinematic topology.

In contrast to these methods, our goal is to encode joint-level features, shape and camera information separately, rather than encoding all the information into a unified image feature vector.

Since we use independent tokens to encode the rotational information of each joint, we can model the inner temporal patterns when each joint rotates over time.

Capturing The Motion Of Every Joint: 3D Hu-Man Pose And Shape Estimation With Indepen-Dent Tokens

Abstract: In this paper we present a novel method to estimate 3D human pose and shape from monocular videos. This task requires directly recovering pixel-alignment 3D human pose and body shape from monocular images or videos, which is challenging due to its inherent ambiguity. To improve precision, existing methods highly rely on the initialized mean pose and shape as prior estimates and parameter regression with an iterative error feedback manner. In addition, video-based approaches model the overall change over the image-level features to temporally enhance the single-frame feature, but fail to capture the rotational motion at the joint level, and cannot guarantee local temporal consistency. To address these issues, we propose a novel Transformer-based model with a design of independent tokens. First, we introduce three types of tokens independent of the image feature: joint rotation tokens, shape token, and camera token. By progressively interacting with image features through Transformer layers, these tokens learn to encode the prior knowledge of human 3D joint rotations, body shape, and position information from large-scale data, and are updated to estimate SMPL parameters conditioned on a given image. Second, benefiting from the proposed token-based representation, we further use a temporal model to focus on capturing the rotational temporal information of each joint, which is empirically conducive to preventing large jitters in local parts. Despite being conceptually simple, the proposed method attains superior performances on the 3DPW and Human3.6M datasets. Using ResNet-50 and Transformer architectures, it obtains 42.0 mm error on the PA-MPJPE metric of the challenging 3DPW, outperforming state-of-the-art counterparts by a large margin. Code will be publicly available 1 .

Learning 3D Human Dynamics From Video

Abstract: From an image of a person in action, we can easily guess the 3D motion of the person in the immediate past and future. This is because we have a mental model of 3D human dynamics that we have acquired from observing visual sequences of humans in motion. We present a framework that can similarly learn a representation of 3D dynamics of humans from video via a simple but effective temporal encoding of image features. At test time, from video, the learned temporal representation give rise to smooth 3D mesh predictions. From a single image, our model can recover the current 3D mesh as well as its 3D past and future motion. Our approach is designed so it can learn from videos with 2D pose annotations in a semi-supervised manner. Though annotated data is always limited, there are millions of videos uploaded daily on the Internet. In this work, we harvest this Internet-scale source of unlabeled data by training our model on unlabeled video with pseudo-ground truth 2D pose obtained from an off-the-shelf 2D pose detector. Our experiments show that adding more videos with pseudo-ground truth 2D pose monotonically improves 3D prediction performance. We evaluate our model on the recent challenging dataset of 3D Poses in the Wild and obtain state-of-the-art performance on the 3D prediction task without any fine-tuning. The project website with video can be found at https://akanazawa.github.io/human_dynamics/.


D. Extending The Method To The Two-Way Distillable Entanglement

However,Ê npt rev (•) is efficiently computable by SDP and tightens the upper bound of the example states illustrated in #REFR , whose details can be found in Appendix C.

It also follows an efficiently computable relaxation asÊ npt

rev (ρ AB ) = [1 − 2 −R max,PPT (ρ AB ) ] i λ i S(B) ψ i , where ω AB = i λ i |ψ i ψ i |

is the spectral decomposition of the PPT-squeezed state ω AB of ρ AB .

In fact,Ê npt rev (•) can be interpreted as an easily computable version of the bound E MP (•) in #OTHEREFR , utilizing the convexity of E D,↔ (ρ AB ) on the convex decomposition of ρ AB into MC states and PPT states.

Since the set of all MC states is not convex, tracking all possible decompositions to compute E MP (•) is generally hard.

We note that R(ρ AB ) ≤ E MP (ρ AB ) ≤ E npt rev (ρ AB ) where R(•) is the Rains bound for the two-way distillable entanglement.

Nevertheless,Ê npt rev (•) connects the reverse max-relative entropy of NPT entanglement with the entanglement of formation, and we believe such connection would shed light on the study of other quantum resource theories as well.

Estimate Distillable Entanglement And Quantum Capacity By Squeezing Useless Entanglement

Abstract: Entanglement distillation is crucial in quantum information processing. But it remains challenging to estimate the distillable entanglement and its closely related essential quantity, the quantum capacity of a noisy quantum channel. In this work, we propose methods for evaluating both quantities by squeezing out useless entanglement within a state or a quantum channel, whose contributions are expected to be ignored for the distillable entanglement or the quantum capacity, respectively. We first consider a general resource measure called the reverse divergence of resources to quantify the minimum divergence between a target state and the set of free states. We then introduce the reverse max-relative entropy of entanglement and apply it to establish efficiently computable upper bounds on the distillable entanglement. We also extend the reverse divergence of resources to quantum channels and derive upper bounds on the quantum capacity. We further apply our method to investigate purifying the maximally entangled states under practical noises, such as depolarizing and amplitude damping noises, and notably establish improvements in estimating the one-way distillable entanglement. Our bounds also offer useful benchmarks for evaluating the quantum capacities of qubit quantum channels of interest, including the Pauli channels and the random mixed unitary channels. CONTENTS I. Introduction A. Background B. Main contributions II. Reverse divergence of resources A. Preliminaries B. Reverse divergence of resources III. Applications on distillable entanglement A. Upper bound on the one-way distillable entanglement B. Continuity bounds of the one-way distillable entanglement C. Examples of less-entangled states D. Extending the method to the two-way distillable entanglement IV. Applications on quantum channel capacity A. Quantum capacity of qubit channels V. Concluding remarks Acknowledgements.

Useful States And Entanglement Distillation

Abstract: We derive general upper bounds on the distillable entanglement of a mixed state under one-way and two-way local operations and classical communication (LOCC). In both cases, the upper bound is based on a convex decomposition of the state into “useful” and “useless” quantum states. By “useful,” we mean a state whose distillable entanglement is non-negative and equal to its coherent information (and thus given by a single-letter, tractable formula). On the other hand, “useless” states are undistillable, i.e., their distillable entanglement is zero. We prove that in both settings, the distillable entanglement is convex on such decompositions. Hence, an upper bound on the distillable entanglement is obtained from the contributions of the useful states alone, being equal to the convex combination of their coherent informations. Optimizing over all such decompositions of the input state yields our upper bound. The useful and useless states are given by degradable and antidegradable states in the one-way LOCC setting, and by maximally correlated and positive partial transpose (PPT) states in the two-way LOCC setting, respectively. We also illustrate how our method can be extended to quantum channels. Interpreting our upper bound as a convex roof extension, we show that it reduces to a particularly simple, non-convex optimization problem for the classes of isotropic states and Werner states. In the one-way LOCC setting, this non-convex optimization yields an upper bound on the quantum capacity of the qubit depolarizing channel that is strictly tighter than previously known bounds for large values of the depolarizing parameter. In the two-way LOCC setting, the non-convex optimization achieves the PPT-relative entropy of entanglement for both isotropic and Werner states.


InfoNCE #REFR (Information Noise-Contrastive Estimation) is based on contrastive learning, which uses noise-contrastive estimation (Noise-Contrastive Estimation, NCE) as a bound to approximate mutual information, and uses neural networks to parameterize critic in NCE, so that it can be used more flexibly.

Where is a multidimensional variable with ( > 2) dimensions, and ( ) obtains the total correlation among all dimensions of .

Since the exact computation of mutual information and total correlation is only tractable for discrete variables, or for a limited family of problems where the probability distributions are known #OTHEREFR .

Therefore, many methods have been proposed to estimate MI based on theoretical upper or lower bounds.

For instance, MINE #OTHEREFR maximizes an implicit function lower bound obtained by Jensen's inequality to estimate the mutual information between two random variables.

The CLUB #OTHEREFR algorithm employs unsupervised learning methods of logarithmic comparison and log-linear models to avoid overfitting and improve the accuracy of estimates.

CLUB uses a technique called Contrastive Log-ratio Upper Bound to calculate the confidence interval and upper bound of the estimated TC value.

This technique is based on the distribution of log comparison values and some statistical assumptions, which can effectively control the error and confidence of the estimate.

The same issue of an intractable computation arises when estimating the TC value of multidimensional variables without knowledge of the ( ) distribution.

In representation learning, a straightforward prior distribution assumption of ( ) is frequently made to handle this issue #OTHEREFR .

Understanding The Limitation Of Total Correlation Estimation Based On Mutual Information Bounds

Abstract: The total correlation(TC) is a crucial index to measure the correlation between marginal distribution in multidimensional random variables, and it is frequently applied as an inductive bias in representation learning. Previous research has shown that the TC value can be estimated using mutual information boundaries through decomposition. However, we found through theoretical derivation and qualitative experiments that due to the use of importance sampling in the decomposition process, the bias of TC value estimated based on MI bounds will be amplified when the proposal distribution in the sampling differs significantly from the target distribution. To reduce estimation bias issues, we propose a TC estimation correction model based on supervised learning, which uses the training iteration loss sequence of the TC estimator based on MI bounds as input features to output the true TC value. Experiments show that our proposed method can improve the accuracy of TC estimation and eliminate the variance generated by the TC estimation process.

Representation Learning With Contrastive Predictive Coding

Abstract: While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.


The VLP model was pre-trained using the Adam optimizer with decoupled weight decay (AdamW), along with a warmup cosine annealing scheduler #REFR ).

The initial learning rate for the warm-up phase was set to 1e−5, which was gradually increased to a maximum learning rate of 1e − 4 before being decayed using the cosine annealing scheduler.

The visual encoder was initialized with self-supervised weights from ImageNet, while the text encoder was trained from scratch.

The model was trained for 15 epochs with a warm-up epoch of 5, and a batch size of 12.

All VLP model training was performed on a GeForce RTX 3090.

For fine-tuning, we used the same optimizer and scheduler, with a learning rate of 1e−4 and a warm-up learning rate of 1e− 5.

Improving Medical Speech-To-Text Accuracy With Vision-Language Pre-Training Model

Abstract: Automatic Speech Recognition (ASR) is a technology that converts spoken words into text, facilitating interaction between humans and machines. One of the most common applications of ASR is Speech-To-Text (STT) technology, which simplifies user workflows by transcribing spoken words into text. In the medical field, STT has the potential to significantly reduce the workload of clinicians who rely on typists to transcribe their voice recordings. However, developing an STT model for the medical domain is challenging due to the lack of sufficient speech and text datasets. To address this issue, we propose a medical-domain text correction method that modifies the output text of a general STT system using the Vision Language Pre-training (VLP) method. VLP combines textual and visual information to correct text based on image knowledge. Our extensive experiments demonstrate that the proposed method offers quantitatively and clinically significant improvements in STT performance in the medical field. We further show that multi-modal understanding of image and text information outperforms single-modal understanding using only text information.

Sgdr: Stochastic Gradient Descent With Warm Restarts

Abstract: Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at https://github.com/loshchil/SGDR


For the text encoder, we choose between bert-base-uncased #REFR and SimCSE Gao et al. (2021) .

Our approach presents one simple but effective way to use that knowledge.

We follow the standard training procedure ( §2.4) and train a CLIP-base model where both of the encoders are initialized randomly, instead of using weights initialized from unimodally pretrained models (DeIT Touvron et al. (2021) and SimCSE Gao et al. (2021) ). We train three models, one for each dataset size. The results can be seen in Fig 6.

Compared to the randomly initialized model, the pretrained model is substantially better across all three datasets and all 3 model sizes.

However, it is likely that the benefit of unimodal pretraining will be diminished as the number of training pairs available for multimodal vision-language pretraining increases, although we do not explore this.

We train LilT-base models with encoders initialized from different kinds of pretraining methods.

For the image encoder, we choose between DeiTTouvron et al. 2021

Contrastive Alignment Of Vision To Language Through Parameter-Efficient Transfer Learn-Ing

Abstract: Contrastive vision-language models (e.g. CLIP) are typically created by updating all the parameters of a vision model and language model through contrastive training. Can such models be created by a small number of parameter updates to an already-trained language model and vision model? The literature describes techniques that can create vision-language models by updating a small number of parameters in a language model, but these require already aligned visual representations and are non-contrastive, hence unusable for latency-sensitive applications such as neural search. We explore the feasibility and benefits of parameter-efficient contrastive vision-language alignment through transfer learning: creating a model such as CLIP by minimally updating an already-trained vision and language model. We find that a minimal set of parameter updates (<7%) can achieve the same performance as full-model training, and updating specific components (<1% of parameters) can match 75% of full-model training. We describe a series of experiments: we show that existing knowledge is conserved more strongly in parameter-efficient training and that parameter-efficient scaling scales with model and dataset size. Where paired-image text data is scarce but strong multilingual language models exist (e.g. low resource languages), parameter-efficient training is even preferable to full-model training. Given a fixed compute budget, parameter-efficient training allows training larger models on the same hardware, achieving equivalent performance in less time. Parameter-efficient training hence constitutes an energyefficient and effective training strategy for contrastive vision-language models that may be preferable to the full-model training paradigm for common use cases. Code and weights at https://github.com/codezakh/LilT.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


In the last decades multilevel Monte Carlo (MLMC) methods have been applied to a plethora of problems in stochastic modelling and uncertainty quantification (see e.g. [5, 11, 14, 19, #REFR ).

The method relies on a hierarchy of approximations arranged as a telescoping sum, resulting in a variance reduction.

In fact, in certain situations the multilevel estimator has asymptotically the same computational complexity as one solve of the deterministic problem on the finest discretization of the hierarchy.

Having a preset finest discretization level the multilevel estimator is a biased estimator.

The continuous level Monte Carlo (CLMC) estimator circumvents this issue by defining the estimator as a stochastic process (see [12] ).

The level of refinement is here given by an exponentially distributed random variable, which in turn means that the computational complexity of a CLMC estimator relies heavily on the concrete samples of this random variable in the simulation.

Quasi Continuous Level Monte Carlo For Random Elliptic Pdes

Abstract: This paper provides a framework in which multilevel Monte Carlo and continuous level Monte Carlo can be compared. In continuous level Monte Carlo the level of refinement is determined by an exponentially distributed random variable, which therefore heavily influences the computational complexity. We propose in this paper a variant of the algorithm, where the exponentially distributed random variable is generated by a quasi Monte Carlo sequence, resulting in a significant variance reduction. In the examples presented the quasi continuous level Monte Carlo algorithm outperforms multilevel and continuous level Monte Carlo by a clear margin.

Further Analysis Of Multilevel Monte Carlo Methods For Elliptic Pdes With Random Coefficients

Abstract: We consider the application of multilevel Monte Carlo methods to elliptic PDEs with random coefficients. We focus on models of the random coefficient that lack uniform ellipticity and boundedness with respect to the random parameter, and that only have limited spatial regularity. We extend the finite element error analysis for this type of equation, carried out recently by Charrier, Scheichl and Teckentrup, to more difficult problems, posed on non--smooth domains and with discontinuities in the coefficient. For this wider class of model problem, we prove convergence of the multilevel Monte Carlo algorithm for estimating any bounded, linear functional and any continuously Fr\'echet differentiable non--linear functional of the solution. We further improve the performance of the multilevel estimator by introducing level dependent truncations of the Karhunen--Lo\`eve expansion of the random coefficient. Numerical results complete the paper.


To validate the effectiveness of our approach, we conduct dialogue-to-video experiments on a benchmark dataset AVSD #REFR .

Such discussion contains subtle information about the video of interest and thus cannot be treated as a plain-text query.

Therefore, to incorporate the conversational information from dialogues, we propose a novel dialogue-to-video retrieval approach.

In our proposed model, we sequentially encode each turn of the dialogue to obtain a dialogue-aware query representation with the purpose of retaining the dialogue information.

Then we calculate the similarity between this dialogue-aware query representation and individual frames in the video in order to obtain a weighted video representation.

Finally, we use the video representation to compute an overall similarity score with the dialogue-aware query.

Experimental results show that our approach achieves significant improvements over previous state-of-the-art models including FiT and ViReD #OTHEREFR .

In this section, we describe how our dialogue-to-video retrieval system works.

Our retrieval system consists of two major components: 1) a temporal-aware video encoder responsible for encoding the image frames in video with temporal information.

2) a dialogue-query encoder responsible for encoding the dialogue query with conversational information.

As shown in Figure 1 , our model receives video-query pairs and produces similarity scores.

Dialogue-To-Video Retrieval

Abstract: Recent years have witnessed an increasing amount of dialogue/conversation on the web especially on social media. That inspires the development of dialogue-based retrieval, in which retrieving videos based on dialogue is of increasing interest for recommendation systems. Different from other video retrieval tasks, dialogue-to-video retrieval uses structured queries in the form of user-generated dialogue as the search descriptor. We present a novel dialogue-to-video retrieval system, incorporating structured conversational information. Experiments conducted on the AVSD dataset show that our proposed approach using plain-text queries improves over the previous counterpart model by 15.8% on R@1. Furthermore, our approach using dialogue as a query, improves retrieval performance by 4.

Audio Visual Scene-Aware Dialog (Avsd) Challenge At Dstc7

Abstract: Scene-aware dialog systems will be able to have conversations with users about the objects and events around them. Progress on such systems can be made by integrating state-of-the-art technologies from multiple research areas including end-to-end dialog systems visual dialog, and video description. We introduce the Audio Visual Scene Aware Dialog (AVSD) challenge and dataset. In this challenge, which is one track of the 7th Dialog System Technology Challenges (DSTC7) workshop1, the task is to build a system that generates responses in a dialog about an input video


The Adaptive Moment Estimation (Adam) #REFR is used as our optimizer with a learning rate of 0.0001 and a batch size of 32.

We utilize the RawNet2 [4] model as the backbone for feature learning.

The loss weight λ is set as 0.5 in the experiment.

To report the detection performance, we calculate the Equal Error Rate (EER) following previous studies #OTHEREFR 29, #OTHEREFR . RawNet2 #OTHEREFR .

The RawNet2 model is based on DNN speaker embedding extraction with the raw waveform as inputs.

This powerful model uses a technique named feature map scaling which scales feature maps similar to squeezeexcitation.

It performed the best in the ASVspoof 2021 Speech Deepfake track.

Ai-Synthesized Voice Detection Using Neural Vocoder Artifacts

Abstract: Advancements in AI-synthesized human voices have created a growing threat of impersonation and disinformation, making it crucial to develop methods to detect synthetic human voices. This study proposes a new approach to identifying synthetic human voices by detecting artifacts of vocoders in audio signals. Most DeepFake audio synthesis models use a neural vocoder, a neural network that generates waveforms from temporal-frequency representations like mel-spectrograms. By identifying neural vocoder processing in audio, we can determine if a sample is synthesized. To detect synthetic human voices, we introduce a multi-task learning framework for a binaryclass RawNet2 model that shares the feature extractor with a vocoder identification module. By treating vocoder identification as a pretext task, we constrain the feature extractor to focus on vocoder artifacts and provide discriminative features for the final binary classifier. Our experiments show that the improved RawNet2 model based on vocoder identification achieves high classification performance on the binary task overall. Codes and data can be found at https://github.com/csun22/Synthetic-Voice-Detection-Vocoder-Artifacts.

Adam: A Method For Stochastic Optimization

Abstract: We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.

learning rate
Adaptive Moment Estimation


Related Works 6.1 Collaborative Filtering

In recent years, this type of global relationships has gradually received more attention and has been incorporated in CF models in the form of interaction graph #REFR .

Since CF can be considered as a task to complete entries in the user-item interaction matrix, Matrix Factorization (MF) #OTHEREFR , as a strategy for matrix completion, naturally becomes the foundation of the mainstream approach in CF.

MF assumes that the user-item interaction matrix is low-rank and can be recovered by learning the embedding vectors of users and items.

Most MF models generate predicted entries by the dot product of user and item embedding vectors, while they can be optimized by minimizing the error of individual entries #OTHEREFR or maximizing the difference between positive and negative samples #OTHEREFR .

They are both widely adopted in the subsequent proposed methods which introduce refined structures like neural networks #OTHEREFR .

These approaches enable a light design by focusing on the modeling of single user-item entry, while neglecting the synergy between different interactions.

With the emerging of Graph Convolutional Networks (GCN), GCN models #OTHEREFR quickly become the state-of-the-art in MF-based models and are continuously improved to achieve advances in efficiency and accuracy #OTHEREFR .

Unlike MF-based models, another class of method implements CF by treating the user's historical interactions as features, and modeling the relationships between items #OTHEREFR .

A classical approach is the linear autoencoder #OTHEREFR , which models an item-item relationship matrix to encode user features.

This idea is then extended by subsequent studies and applied to nonlinear denoising autoencoders #OTHEREFR and variational autoencoders #OTHEREFR .

A recent work #OTHEREFR considers CF in terms of graph signal processing and proposes a framework for signal-based models, which can incorporate the linear autoencoders and the ideal case of MF and GCN models.

Collaborative Residual Metric Learning

Abstract: In collaborative filtering, distance metric learning has been applied to matrix factorization techniques with promising results. However, matrix factorization lacks the ability of capturing collaborative information, which has been remarked by recent works and improved by interpreting user interactions as signals. This paper aims to find out how metric learning connect to these signal-based models. By adopting a generalized distance metric, we discovered that in signal-based models, it is easier to estimate the residual of distances, which refers to the difference between the distances from a user to a target item and another item, rather than estimating the distances themselves. Further analysis also uncovers a link between the normalization strength of interaction signals and the novelty of recommendation, which has been overlooked by existing studies. Based on the above findings, we propose a novel model to learn a generalized distance user-item distance metric to capture user preference in interaction signals by modeling the residuals of distance. The proposed CoRML model is then further improved in training efficiency by a newly introduced approximated ranking weight. Extensive experiments conducted on 4 public datasets demonstrate the superior performance of CoRML compared to the state-of-the-art baselines in collaborative filtering, along with high efficiency and the ability of providing novelty-promoted recommendations, shedding new light on the study of metric learning-based recommender systems. CCS CONCEPTS • Information systems → Recommender systems; Collaborative filtering.

Spectral Collaborative Filtering

Abstract: Despite the popularity of Collaborative Filtering (CF), CF-based methods are haunted by the cold-start problem, which has a significantly negative impact on users' experiences with Recommender Systems (RS). In this paper, to overcome the aforementioned drawback, we first formulate the relationships between users and items as a bipartite graph. Then, we propose a new spectral convolution operation directly performing in the spectral domain, where not only the proximity information of a graph but also the connectivity information hidden in the graph are revealed. With the proposed spectral convolution operation, we build a deep recommendation model called Spectral Collaborative Filtering (SpectralCF). Benefiting from the rich information of connectivity existing in the spectral domain, SpectralCF is capable of discovering deep connections between users and items and therefore, alleviates the cold-start problem for CF. To the best of our knowledge, SpectralCF is the first CF-based method directly learning from the spectral domains of user-item bipartite graphs. We apply our method on several standard datasets. It is shown that SpectralCF significantly out-performs state-of-the-art models. Code and data are available at https://github.com/lzheng21/SpectralCF.


To be specific, for modeling hierarchies in natural language, we share similar intuitions with the previous studies on unsupervised grammar induction, which aim at unsupervised hierarchical mining #REFR Drozdov et al., 2019) .

To this end, we introduce hierarchy-aware attention into CLIP, denoted as HiCLIP.

Hierarchy-aware attention applies an attention mask to the conventional attention mechanism to indicate the tendency to merge certain vision patches and language tokens into groups because they are spatially and semantically or visually similar.

We generalize hierarchy-aware attention to both images and texts, where its mask is obtained by first calculating the neighbouring affinity score among adjacent patches or tokens, and then propagating the scores across any given patch or token pairs.

In addition, we formulate the affinity score with an increasing trend as the layer gets deeper to ensure the merged groups remains the same.

In this way, we progressively aggregate hierarchies in a layer-by-layer manner for both images and texts.

Tree Transformer proposes a similar modified attention mechanism which is essentially a special case of hierarchy-aware attention, where the attention mask is instantiated as constituent prior to encourage the merging of semantically similar tokens.

Capturing hierarchies in visual contents is more challenging, because spatial correlation should also be considered in addition to visual similarities.

Therefore, we extend the hierarchy-aware attention to Vision Transformers #OTHEREFR by creating a Group Transformer to progressively aggregate image patches into semantic groups until all patches are merged in one common group which is the original image.

Different from the 1D scenario in Tree Transformer, the neighboring affinity score is computed among the four adjacent neighbors of each image patch ( Fig. 1 (a) ).

Afterwards, we propagate neighboring affinity scores by comparing two special paths connecting image patches on the 2D grid graph.

Hiclip: Contrastive Language-Image Pre-Training With Hierarchy-Aware Attention

Abstract: The success of large-scale contrastive vision-language pretraining (CLIP) has benefited both visual recognition and multimodal content understanding. The concise design brings CLIP the advantage in inference efficiency against other visionlanguage models with heavier cross-attention fusion layers, making it a popular choice for a wide spectrum of downstream tasks. However, CLIP does not explicitly capture the hierarchical nature of high-level and fine-grained semantics conveyed in images and texts, which is arguably critical to vision-language understanding and reasoning. To this end, we equip both the visual and language branches in CLIP with hierarchy-aware attentions, namely Hierarchy-aware CLIP (HiCLIP), to progressively discover semantic hierarchies layer-by-layer from both images and texts in an unsupervised manner. As a result, such hierarchical aggregation significantly improves the cross-modal alignment. To demonstrate the advantages of HiCLIP, we conduct qualitative analysis on its unsupervised hierarchy induction during inference, as well as extensive quantitative experiments on both visual recognition and vision-language downstream tasks. 1 * This work was conducted while interning at ByteDance.

Visually Grounded Neural Syntax Acquisition

Abstract: We present the Visually Grounded Neural Syntax Learner (VG-NSL), an approach for learning syntactic representations and structures without any explicit supervision. The model learns by looking at natural images and reading paired captions. VG-NSL generates constituency parse trees of texts, recursively composes representations for constituents, and matches them with images. We define concreteness of constituents by their matching scores with images, and use it to guide the parsing of text. Experiments on the MSCOCO data set show that VG-NSL outperforms various unsupervised parsing approaches that do not use visual grounding, in terms of F1 scores against gold parse trees. We find that VGNSL is much more stable with respect to the choice of random initialization and the amount of training data. We also find that the concreteness acquired by VG-NSL correlates well with a similar measure defined by linguists. Finally, we also apply VG-NSL to multiple languages in the Multi30K data set, showing that our model consistently outperforms prior unsupervised approaches.


The authors propose the sliced Wasserstein discrepancy, which considers the properties of the underlying geometry of probability space and thus im-proves upon the L1 norm used in #REFR . Further follow-up work is presented in Li et al.

In the third step, the feature extractor is then optimized to minimize the MCD for the target domain.

This causes those samples from the target domain far away from the source domain distribution to move closer to the source domain distribution (here, the support of the source domain is given, and the two classifiers agree).

The three steps are iterated, which results in an adversarial optimization.

#OTHEREFR advance this work by introducing an improved way of computing the discrepancy between the classifier probability outputs.

#OTHEREFR where two classifiers are trained on the source domain while also updating the feature generator.

Then they maximize the classifier discrepancy on the target domain while ensuring the source domain classification stays the same.

In the final stage, they train the feature generator to minimize the classifier discrepancy, pushing the target domain data to the statistical support of the source domain.

Other methods for distribution divergence minimization: MMD #OTHEREFR uses the soft paste algorithm, combining two images by a weighted overlay (see Section III-B).

A reference source domain image is pasted into a target and source domain image.

Survey On Unsupervised Domain Adaptation For Semantic Segmentation For Visual Perception In Automated Driving

Abstract: The research leading to these results is funded by the German Federal Ministry for Economic Affairs and Energy within the project "KI Delta Learning" (Förderkennzeichen 19A19013C and 19A19013K). The authors would like to thank the consortium for the successful cooperation.

Maximum Classifier Discrepancy For Unsupervised Domain Adaptation

Abstract: In this work, we present a method for unsupervised domain adaptation (UDA), where we aim to transfer knowledge from a label-rich domain (i.e., a source domain) to an unlabeled domain (i.e., a target domain). Many adversarial learning methods have been proposed for this task. These methods train domain classifier networks (i.e., a discriminator) to discriminate distinguish the features as either a source or target and train a feature generator network to mimic the discriminator.However, the domain classifier only tries to distinguish the features as a source or target and thus does not consider task-specific decision boundaries between classes. Therefore, a trained generator can generate ambiguous features near class boundaries. ::: To solve the problem, we propose a new approach that attempts to align distributions of source and target by utilizing the task-specific decision boundaries. We propose to utilize task-specific classifiers as discriminators that try to detect target samples that are far from the support of the source. A feature generator learns to generate target features inside the support to fool the classifiers. Since the generator uses feedback from task-specific classifiers, it avoids generating target features near class boundaries. Our method outperforms other methods on several datasets of image classification and semantic segmentation.


First, the standard indicator and list view exemplified by Google Scholar, as shown in Figure 1 (a), provides key bibliographic indicators (# of citations, h-index #REFR , etc.) as an overview of the scholar.

Commonly, scholar profiling is composed of two steps: 1) accurate extraction of 360-degree academic demographics of a scholar from the web, which is normally considered an information retrieval task; and 2) appropriate aggregation, analysis, and representation of the extracted academic data, which is largely a data mining task.

This work focuses on the latter step and formulates the impactoriented scholar profiling problem -how to arrange a scholar's academic data to best represent his/her scientific impact.

Here the impact is defined as the breadth and depth of one's scientific contribution, as well as their community recognition.

The problem is closely related to important applications of scholar profiles, such as serving possible references for academic award selection and tenure evaluation #OTHEREFR , and predicting one's future academic output [20] .

Existing scholar profiling websites generally design two types of views.

The view is complemented with multiple lists as the context (publications, co-authors, etc.).

The indicators are believed to be helpful for scholar ranking, though the supplemented paper/coauthor lists are normally unprocessed.

There is a missed opportunity to exploit the structure within these lists for impact-oriented contextual tasks beyond ranking, such as reasoning and understanding of scholar profiles.

Second, scholarly websites from academics often exhibit various bibliometric networks.

These networks somehow mitigate the drawback of indicator and list by illustrating the relational context of a scholar.

Impact-Oriented Contextual Scholar Profiling Using Self-Citation Graphs

Abstract: Quantitatively profiling a scholar's scientific impact is important to modern research society. Current practices with bibliometric indicators (e.g., h-index), lists, and networks perform well at scholar ranking, but do not provide structured context for scholar-centric, analytical tasks such as profile reasoning and understanding. This work presents GeneticFlow (GF), a suite of novel graph-based scholar profiles that fulfill three essential requirements: structured-context, scholar-centric, and evolution-rich. We propose a framework to compute GF over large-scale academic data sources with millions of scholars. The framework encompasses a new unsupervised advisoradvisee detection algorithm, a well-engineered citation type classifier using interpretable features, and a fine-tuned graph neural network (GNN) model. Evaluations are conducted on the real-world task of scientific award inference. Experiment outcomes show that the F1 score of best GF profile significantly outperforms alternative methods of impact indicators and bibliometric networks in all the 6 computer science fields considered. Moreover, the core GF profiles, with 63.6%∼66.5% nodes and 12.5%∼29.9% edges of the full profile, still significantly outrun existing methods in 5 out of 6 fields studied. Visualization of GF profiling result also reveals human explainable patterns for high-impact scholars.

An Index To Quantify An Individual'S Scientific Research Output

Abstract: I propose the index $h$, defined as the number of papers with citation number higher or equal to $h$, as a useful index to characterize the scientific output of a researcher.


Usually, generative models attempt to learn the reconstruction of the data and use this reconstruction to identify anomalous samples #REFR .

Deep support vector data description (DSVDD) finds a hypersphere to enclose the representation of normal samples #OTHEREFR .

Liu and Gryllias constructed frequency domain features using cyclic spectral analysis and applied them in the support vector data description (SVDD) framework.

This method has been proven robust against outliers and can achieve a high detection rate for detecting anomalies #OTHEREFR .

In #OTHEREFR , researchers presented a new approach to identify imagery anomalies by training the model on normal images altered by geometric transformation.

In this model, the classifier calculates the anomaly score using softmax statistics.

For instance, auto-encoders model the normal data distribution, and the reconstruction error is used as the anomaly score #OTHEREFR .

Deep structured energy-based models (DSEBMs) learn an energy-based model and map each sample to an energy score #OTHEREFR .

Deep autoencoding Gaussian mixture model (DAGMM) estimates a mixed Gaussian distribution by using an encoder for normal samples #OTHEREFR .

A recent line of work on anomaly detection has focused on adversarial neural networks.

For example, this structure has been used to identify anomalies in medical images #OTHEREFR .

Regularized Complete Cycle Consistent Gan For Anomaly Detection

Abstract: This study presents an adversarial method for anomaly detection in real-world applications, leveraging the power of generative adversarial neural networks (GANs) through cycle consistency in reconstruction error. Previous methods suffer from the high variance between class-wise accuracy which leads to not being applicable for all types of anomalies. The proposed method named RCALAD tries to solve this problem by introducing a novel discriminator to the structure, which results in a more efficient training process. Additionally, RCALAD employs a supplementary distribution in the input space to steer reconstructions toward the normal data distribution, effectively separating anomalous samples from their reconstructions and facilitating more accurate anomaly detection. To further enhance the performance of the model, two novel anomaly scores are introduced. The proposed model has been thoroughly evaluated through extensive experiments on six various datasets, yielding results that demonstrate its superiority over existing state-of-the-art models. The code is readily available to the research community at https://github.com/zahraDehghanian97/RCALAD.

Regularized Cycle Consistent Generative Adversarial Network For Anomaly Detection

Abstract: In this paper, we investigate algorithms for anomaly detection. Previous anomaly detection methods focus on modeling the distribution of non-anomalous data provided during training. However, this does not necessarily ensure the correct detection of anomalous data. We propose a new Regularized Cycle Consistent Generative Adversarial Network (RCGAN) in which deep neural networks are adversarially trained to better recognize anomalous samples. This approach is based on leveraging a penalty distribution with a new definition of the loss function and novel use of discriminator networks. It is based on a solid mathematical foundation, and proofs show that our approach has stronger guarantees for detecting anomalous examples compared to the current state-of-the-art. Experimental results on both real-world and synthetic data show that our model leads to significant and consistent improvements on previous anomaly detection benchmarks. Notably, RCGAN improves on the state-of-the-art on the KDDCUP, Arrhythmia, Thyroid, Musk and CIFAR10 datasets.


NuScenes #REFR presents a greater challenge for 3D SOT task than KITTI due to its larger data volumes and lower frequency for annotated frames (2Hz for NuScenes v.s. 10Hz for KITTI and WOD).

However, only MBP-Track can accurately track the target after the occlusion disappears, owing to the sufficient use of temporal information.

To demonstrate the generalization ability of our proposed MBPTrack, we evaluate the KITTI pretrained models on WOD #OTHEREFR , following previous work #OTHEREFR .

The corresponding categories between KITTI and WOD datasets are Car→Vehicle and Pedestrian→Pedestrian. The experimental results, as presented in Tab.

2, indicate that MBP-Track yields competitive or better tracking results than other methods under different levels of sparsity.

In conclusion, our proposed method not only precisely tracks targets of all sizes but also generalizes well to unseen scenarios.

We conduct a comparison of our approach with previous methods on the NuScenes dataset following M2-Track #OTHEREFR . As shown in Tab.

3, our method achieves a consistent and large performance gain compared with the previous state-of-the-art method, M2-Track.

Leveraging the rich temporal and spatial information contained in the historical frames, MBPTrack exhibits superior performance over methods that only consider two frames when large appearance variation occurs between them. Fig.

4 shows the model complexity and average inference time of different components in the Car category on KITTI.

Our experiments are conducted on a single NVIDIA RTX 3090.

Mbptrack: Improving 3D Point Cloud Tracking With Memory Networks And Box Priors

Abstract: 3D single object tracking has been a crucial problem for decades with numerous applications such as autonomous driving. Despite its wide-ranging use, this task remains challenging due to the significant appearance variation caused by occlusion and size differences among tracked targets. To address these issues, we present MBPTrack, which adopts a Memory mechanism to utilize past information and formulates localization in a coarse-to-fine scheme using Box Priors given in the first frame. Specifically, past frames with targetness masks serve as an external memory, and a transformer-based module propagates tracked target cues from the memory to the current frame. To precisely localize objects of all sizes, MBPTrack first predicts the target center via Hough voting. By leveraging box priors given in the first frame, we adaptively sample reference points around the target center that roughly cover the target of different sizes. Then, we obtain dense feature maps by aggregating point features into the reference points, where localization can be performed more effectively. Extensive experiments demonstrate that MBPTrack achieves state-of-the-art performance on KITTI, nuScenes and Waymo Open Dataset, while running at 50 FPS on a single RTX3090 GPU.

Nuscenes: A Multimodal Dataset For Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image-based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first published dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online at this http URL.