In this paper, they presented a stochastic optimization toolchain along with sensitivity analyses for choosing the optimal hyperparameters, and they solved the optimization problem by using a single-objective optimization task as well as a greedy algorithm, a genetic algorithm (GA), and a particle swarm optimization (PSO) algorithm #REFR .

Dubey #OTHEREFR introduced a public transportation decision support system for short-term and long-term prediction of arrival bus times.

This study used the real-world historical data of two Nashville bus system routes.

The approach of this research combined the clustering analysis and Kalman filters with a shared route segment model to produce more accurate arrival time predictions and, based on their results, compared to the basic arrival time prediction model that Nashville MTA was using, their system reduced arrival time prediction errors by 25% on average when predicting the arrival delay an hour ahead and 47% when predicting within a 15-minute future time window #OTHEREFR . S. Basak, F. Sun, S. Sengupta, and A.

Dubey have conducted a similar study #OTHEREFR , using unsupervised clustering mechanisms to optimize transit on-time performance.

As a local case study, they analyzed the monthly and seasonal delays of the Nashville metro region and clustered months with similar patterns.

According to the newest research in #OTHEREFR , dynamic datadriven application systems (DDDAS) that use real-time sensors and a data-driven decision support system can provide online model learning and multi-time-scale analytics to enhance the intelligence of the system.

As part of their study, the authors analyzed an online bus arrival prediction system in Nashville using historical and real-time streaming data, which can be packaged as modular, distributed, and resilient micro-services.

The long-term delay analysis service excludes noise from outliers in historical data to identify delay patterns associated with different hours, days, and seasons for specific time points and route segments.

City planners can use the feedback data generated by these analytics services to improve bus schedules and increase rider satisfaction #OTHEREFR .

In addition, another study by S. Nannapaneni and A.

A Novel Neural Network Approach For Predicting The Arrival Time Of Buses For Smart On-Demand Public Transit

Abstract: Among the major public transportation systems in cities, bus transit has its problems, including more accuracy and reliability when estimating the bus arrival time for riders. This can lead to delays and decreased ridership, especially in cities where public transportation is heavily relied upon. A common issue is that the arrival times of buses do not match the schedules, resulting in latency for fixed schedules. According to the study in this paper on New York City bus data, there is an average delay of around eight minutes or 491 seconds mismatch between the bus arrivals and the actual scheduled time. This research paper presents a novel AI-based data-driven approach for estimating the arrival times of buses at each transit point (station). Our approach is based on a fully connected neural network and can predict the arrival time collectively across all bus lines in large metropolitan areas. Our neural-net data-driven approach provides a new way to estimate the arrival time of the buses, which can lead to a more efficient and smarter way to bring the bus transit to the general public. Our evaluation of the network bus system with more than 200 bus lines, and 2 million data points, demonstrates less than 40 seconds of estimated error for arrival times. The inference time per each validation set data point is less than 0.006 ms.

Data-Driven Optimization Of Public Transit Schedule

Abstract: Bus transit systems are the backbone of public transportation in the United States. An important indicator of the quality of service in such infrastructures is on-time performance at stops, with published transit schedules playing an integral role governing the level of success of the service. However there are relatively few optimization architectures leveraging stochastic search that focus on optimizing bus timetables with the objective of maximizing probability of bus arrivals at timepoints with delays within desired on-time ranges. In addition to this, there is a lack of substantial research considering monthly and seasonal variations of delay patterns integrated with such optimization strategies. To address these, this paper makes the following contributions to the corpus of studies on transit on-time performance optimization: (a) an unsupervised clustering mechanism is presented which groups months with similar seasonal delay patterns, (b) the problem is formulated as a single-objective optimization task and a greedy algorithm, a genetic algorithm (GA) as well as a particle swarm optimization (PSO) algorithm are employed to solve it, (c) a detailed discussion on empirical results comparing the algorithms are provided and sensitivity analysis on hyper-parameters of the heuristics are presented along with execution times, which will help practitioners looking at similar problems. The analyses conducted are insightful in the local context of improving public transit scheduling in the Nashville metro region as well as informative from a global perspective as an elaborate case study which builds upon the growing corpus of empirical studies using nature-inspired approaches to transit schedule optimization.

optimal hyperparameters
particle swarm optimization


The details of how it is implemented can be found under Algorithm A.1 of #REFR , but the main idea is to successively compute the distribution of the underlying state of H after each sample via Bayesian updates.

The Forward Algorithm is prevalent in the preprocessing, compression and decompression algorithms.

Therefore, we provide an overview of what the algorithm does and how it works in this subsection.

Given a Markov source H with ℓ states, let the first j samples be Y 1 , . . . , Y j ∼ H n . For a setting y 1 , . . .

, y j−1 , the Forward Algorithm computes the distribution of Y n given that (Y 1 , . . . , Y j−1 ) = (y 1 , . . . , y j−1 ).

Each update takes time O(ℓ 2 ) (since there are roughly this many possible transitions to consider), and we must do j of these updates, for a total runtime of O(jℓ 2 ). This is implemented using dynamic programming.

Improving The Runtime Of Algorithmic Polarization Of Hidden Markov Models

Abstract: We improve the runtime of the linear compression scheme for hidden Markov sources presented in a 2018 paper of Guruswami, Nakkiran, and Sudan. Under the previous scheme, compressing a message of length n takes O(n log n) runtime, and decompressing takes O(n 1+δ) runtime for any fixed δ > 0. We present how to improve the runtime of the decoding scheme to O(n log n) by caching intermediate results to avoid repeating computation.

Algorithmic Polarization For Hidden Markov Models

Abstract: Using a mild variant of polar codes we design linear compression schemes compressing Hidden Markov sources (where the source is a Markov chain, but whose state is not necessarily observable from its output), and to decode from Hidden Markov channels (where the channel has a state and the error introduced depends on the state). We give the first polynomial time algorithms that manage to compress and decompress (or encode and decode) at input lengths that are polynomial $\it{both}$ in the gap to capacity and the mixing time of the Markov chain. Prior work achieved capacity only asymptotically in the limit of large lengths, and polynomial bounds were not available with respect to either the gap to capacity or mixing time. Our results operate in the setting where the source (or the channel) is $\it{known}$. If the source is $\it{unknown}$ then compression at such short lengths would lead to effective algorithms for learning parity with noise -- thus our results are the first to suggest a separation between the complexity of the problem when the source is known versus when it is unknown.


For the special case of additive valuations with binary marginals, the algorithm of Aziz #REFR improves upon the guarantees of Aleksandrov et al.

#OTHEREFR gave a randomized polynomial-time algorithm that outputs an EF1 allocation, while being envy-free ex-ante.

Ensuing work by Aziz #OTHEREFR showed that there exists a similar randomized algorithm that additionally implements the well-known Probabilistic Serial fractional outcome described by Bogomolnaia and Moulin #OTHEREFR .

This randomized algorithm also preserves a weak notion of efficiency. Babaioff et al.

#OTHEREFR also study the case of additive valuations and find a distribution over ex-post proportional up to one item and #OTHEREFR 2 -MMS allocations that is ex-ante proportional, i.e., the expected value of each agent's bundle is at least a 1 n -fraction of her value for the set of all items.

Best-of-both-worlds results have also been analyzed for other settings, including the case where the agents have binary marginals, and the case of additive valuations with arbitrary entitlements.

#OTHEREFR and is group-strategyproof, ex-ante fractionally-PO and envy-free, and ex-post fractionally PO and EF1.

In a similar vein, for additive valuations with binary marginals, Halpern et al.

#OTHEREFR independently showed that there is a distribution over expost Nash-welfare-maximizing allocations that also ex-ante maximizes the fractional Nash welfare, implying the same fairness guarantees as Aziz #OTHEREFR for this setting. For matroid rank valuations, Babaioff et al.

#OTHEREFR present a randomized truthful mechanism that is ex-ante envy-free and ex-post Lorenz dominating (and thus ex-post Nash-welfare-maximizing and EFX for this specific class).

For the case of agents with arbitrary entitlements, both Hoefer et al. #OTHEREFR and Aziz et al.

Breaking The Envy Cycle: Best-Of-Both-Worlds Guarantees For Subadditive Valuations

Abstract: We study best-of-both-worlds guarantees for the fair division of indivisible items among agents with subadditive valuations. Our main result establishes the existence of a random allocation that is simultaneously ex-ante 1 2-envy-free, ex-post 1 2-EFX and ex-post EF1, for every instance with subadditive valuations. We achieve this result by a novel polynomial-time algorithm that randomizes the well-established envy cycles procedure in a way that provides ex-ante fairness. Notably, this is the first best-of-both-worlds fairness guarantee for subadditive valuations, even when considering only EF1 without EFX.

Simultaneously Achieving Ex-Ante And Ex-Post Fairness

Abstract: We present a polynomial-time algorithm that computes an ex-ante envy-free lottery over envy-free up to one item (EF1) deterministic allocations. It has the following advantages over a recently proposed algorithm: it does not rely on the linear programming machinery including separation oracles; it is SD-efficient (both ex-ante and ex-post); and the ex-ante outcome is equivalent to the outcome returned by the well-known probabilistic serial rule. As a result, we answer a question raised by Freeman, Shah, and Vaish (2020) whether the outcome of the probabilistic serial rule can be implemented by ex-post EF1 allocations. In the light of a couple of impossibility results that we prove, our algorithm can be viewed as satisfying a maximal set of properties. Under binary utilities, our algorithm is also ex-ante group-strategyproof and ex-ante Pareto optimal. Finally, we also show that checking whether a given random allocation can be implemented by a lottery over EF1 and Pareto optimal allocations is NP-hard.


Note that #REFR was designed for trajectory optimization, and we slightly reformulate it to conduct path smoothing.

The output of Algorithm 2 consists of paths originating

; 2 L ← ∅; 3 foreach G z of G do 4 D ← ∅; 5 l init ← InitialGuessSearch(G z ) ; 6 {s i |i = 1, ..., N } ← StationSampling(l init ); 7 for i = 1, ..., N do 8 D i ← LateralClustering(s i ); 9 D ← D ∪ D i 10 end 11 L z ← DPSearch(D, {s i |i = 1, ..., N }, G z ); 12 L z ← NonMaximumSuppress(L z ); 13 L ← L ∪ L z ; 14 end 15 return L;

Due to the discretization of the grid, the path is not smooth enough for control.

To this end, we utilize a local path smoothing #OTHEREFR based on Quadratic Programming (QP) to smooth the path.

An example of the path smoothing process is given in Fig. 4d .

Flowmap: Path Generation For Automated Vehicles In Open Space Using Traffic Flow

Abstract: There is extensive literature on perceiving road structures by fusing various sensor inputs such as lidar point clouds and camera images using deep neural nets. Leveraging the latest advance of neural architects (such as transformers) and bird-eye-view (BEV) representation, the road cognition accuracy keeps improving. However, how to cognize the "road" for automated vehicles where there is no well-defined "roads" remains an open problem. For example, how to find paths inside intersections without HD maps is hard since there is neither an explicit definition for "roads" nor explicit features such as lane markings. The idea of this paper comes from a proverb: it becomes a way when people walk on it. Although there are no "roads" from sensor readings, there are "roads" from tracks of other vehicles. In this paper, we propose FlowMap, a path generation framework for automated vehicles based on traffic flows. FlowMap is built by extending our previous work RoadMap [1], a lightweight semantic map, with an additional traffic flow layer. A path generation algorithm on traffic flow fields (TFFs) is proposed to generate human-like paths. The proposed framework is validated using real-world driving data and is amenable to generating paths for super complicated intersections without using HD maps.

Safe Trajectory Generation For Complex Urban Environments Using Spatio-Temporal Semantic Corridor

Abstract: Planning safe trajectories for autonomous vehicles in complex urban environments is challenging since there are numerous semantic elements (such as dynamic agents, traffic lights, and speed limits) to consider. These semantic elements may have different mathematical descriptions, such as obstacle, constraint, and cost. It is non-trivial to tune the effects from different combinations of semantic elements for a stable and generalizable behavior. In this letter, we propose a novel unified spatio-temporal semantic corridor (SSC) structure, which provides a level of abstraction for different types of semantic elements. The SSC consists of a series of mutually connected collision-free cubes with dynamical constraints posed by the semantic elements in the spatio-temporal domain. The trajectory generation problem then boils down to a general quadratic programming formulation. Thanks to the unified SSC representation, our framework can generalize to any combination of semantic elements. Moreover, our formulation provides a theoretical guarantee that the entire trajectory is safe and constraint-satisfied, by using the convex hull and hodograph properties of piecewise Bezier curve parameterization. We also release the code of our method to accommodate benchmarking.


A model is called perfect calibrated if the predictive confidence of the model represents a good approximation of its actual probability of correctness #REFR .

With the development of deep neural networks, great progress has been made in image classification.

In addition to performance, the uncertainty estimate of a given model is also receiving increasing attention, as the confidence of a model is expected to accurately reflect its performance. † Corresponding author.

Model calibration is particularly important in safety-critical applications, such as autonomous driving, medical diagnosis, and robotics #OTHEREFR .

For example, if a prediction with low confidence is more likely to be wrong, we can take countermeasures to avoid unknown risks.

Most existing calibration techniques assume that the distribution of training data is balanced, i.e., each class has a similar number of training instances, so that each class is treated equally #OTHEREFR . As shown in Fig.

1 , the traditional calibration pipeline uses a balanced training set to train the classification model and a balanced validation set to obtain the calibration model, respectively.

The target test set is in the same distribution as the training/validation set.

Transfer Knowledge From Head To Tail: Uncertainty Calibration Under Long-Tailed Distribution

Abstract: How to estimate the uncertainty of a given model is a crucial problem. Current calibration techniques treat different classes equally and thus implicitly assume that the distribution of training data is balanced, but ignore the fact that real-world data often follows a long-tailed distribution. In this paper, we explore the problem of calibrating the model trained from a long-tailed distribution. Due to the difference between the imbalanced training distribution and balanced test distribution, existing calibration methods such as temperature scaling can not generalize well to this problem. Specific calibration methods for domain adaptation are also not applicable because they rely on unlabeled target domain instances which are not available. Models trained from a long-tailed distribution tend to be more overconfident to head classes. To this end, we propose a novel knowledge-transferring-based calibration method by estimating the importance weights for samples of tail classes to realize long-tailed calibration. Our method models the distribution of each class as a Gaussian distribution and views the source statistics of head classes as a prior to calibrate the target distributions of tail classes. We adaptively transfer knowledge from head classes to get the target probability density of tail classes. The importance weight is estimated by the ratio of the target probability density over the source probability density. Extensive experiments on CIFAR-10-LT, MNIST-LT, CIFAR-100-LT, and ImageNet-LT datasets demonstrate the effectiveness of our method.

On Calibration Of Modern Neural Networks

Abstract: Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.


Silhouette-based methods GaitSet #REFR aggregates temporal information in silhouette sequences using a statistical function to adapt to different frame rates.

Gait recognition methods can be classified into two main categories: appearance-based #OTHEREFR and modelbased methods #OTHEREFR depending on the type of data input.

Appearance-based methods usually rely on silhouette sequences #OTHEREFR or their transformation, such as Gait Energy Image (GEI) #OTHEREFR and Chrono-Gait Image (CGI) #OTHEREFR .

Model-based methods, on the other hand, represent the human body as mesh #OTHEREFR or a set of keypoints #OTHEREFR .

GaitPart #OTHEREFR uses Focal Convolutional Layer to extract fine-grained local features and Micro-motion Template Builder with different window sizes to extract local temporal information.

GaitGL #OTHEREFR uses 3D Convolution to extract global and local spatiotemporal information.

LagrangeGait #OTHEREFR adds a local motion extractor and a viewpoint branch based on GaitGL to get more discriminative local temporal information.

Pose-based methods PoseGait #OTHEREFR employs 3D pose information to generate multi-feature vectors and uses a CNN to extract the gait information in both spatial and temporal dimensions.

GaitGraph #OTHEREFR and GaitGraph2 #OTHEREFR adopt Graph Convolutional Network for gait recognition, treating keypoints as nodes and limbs as edges to form a topology graph.

Gpgait: Generalized Pose-Based Gait Recognition

Abstract: Recent works on pose-based gait recognition have demonstrated the potential of using such simple information to achieve results comparable to silhouette-based methods. However, the generalization ability of pose-based methods on different datasets is undesirably inferior to that of silhouette-based ones, which has received little attention but hinders the application of these methods in real-world scenarios. To improve the generalization ability of pose-based methods across datasets, we propose a Generalized Pose-based Gait recognition (GPGait) framework. First, a Human-Oriented Transformation (HOT) and a series of Human-Oriented Descriptors (HOD) are proposed to obtain a unified pose representation with discriminative multi-features. Then, given the slight variations in the unified representation after HOT and HOD, it becomes crucial for the network to extract local-global relationships between the keypoints. To this end, a Part-Aware Graph Convolutional Network (PAGCN) is proposed to enable efficient graph partition and local-global spatial feature extraction. Experiments on four public gait recognition datasets, CASIA-B, OUMVLP-Pose, Gait3D and GREW, show that our model demonstrates better and more stable cross-domain capabilities compared to existing skeletonbased methods, achieving comparable recognition results to silhouette-based ones. The code will be released.

Gaitset: Regarding Gait As A Set For Cross-View Gait Recognition

Abstract: As a unique biometric feature that can be recognized at a distance, gait has broad applications in crime prevention, forensic identification and social security. To portray a gait, existing gait recognition methods utilize either a gait template, where temporal information is hard to preserve, or a gait sequence, which must keep unnecessary sequential constraints and thus loses the flexibility of gait recognition. In this paper we present a novel perspective, where a gait is regarded as a set consisting of independent frames. We propose a new network named GaitSet to learn identity information from the set. Based on the set perspective, our method is immune to permutation of frames, and can naturally integrate frames from different videos which have been filmed under different scenarios, such as diverse viewing angles, different clothes/carrying conditions. Experiments show that under normal walking conditions, our single-model method achieves an average rank-1 accuracy of 95.0% on the CASIA-B gait dataset and an 87.1% accuracy on the OU-MVLP gait dataset. These results represent new state-of-the-art recognition accuracy. On various complex scenarios, our model exhibits a significant level of robustness. It achieves accuracies of 87.2% and 70.4% on CASIA-B under bag-carrying and coat-wearing walking conditions, respectively. These outperform the existing best methods by a large margin. The method presented can also achieve a satisfactory accuracy with a small number of frames in a test sample, e.g., 82.5% on CASIA-B with only 7 frames. The source code has been released at https://github.com/AbnerHqC/GaitSet.


Processing to remove location and orientation variability has been used in prior work #REFR , and to save space we present our version of normalization in the Appendix.

Before training, temporal windows of body pose data are normalized by centering, rotating and scale normalizing the data values.

Centering and rotation use a body-specific procedure to make the 3D point data invariant to the body's location and facing direction in the capture space.

A Light-Weight Contrastive Approach For Aligning Human Pose Sequences

Abstract: We present a simple unsupervised method for learning an encoder mapping short 3D pose sequences into embedding vectors suitable for sequence-to-sequence alignment by dynamic time warping. Training samples consist of temporal windows of frames containing 3D body points such as mocap markers or skeleton joints. A lightweight , 3layer encoder is trained using a contrastive loss function that encourages embedding vectors of augmented sample pairs to have cosine similarity 1, and similarity 0 with all other samples in a minibatch. When multiple scripted training sequences are available, temporal alignments inferred from an initial round of training are harvested to extract additional, cross-performance match pairs for a second phase of training to refine the encoder. In addition to being simple, the proposed method is fast to train, making it easy to adapt to new data using different marker sets or skeletal joint layouts. Experimental results illustrate ease of use, transferability, and utility of the learned embeddings for comparing and analyzing human behavior sequences.

Predict&Cluster: Unsupervised Skeleton Based Action Recognition

Abstract: We propose a novel system for unsupervised skeleton-based action recognition. Given inputs of body keypoints sequences obtained during various movements, our system associates the sequences with actions. Our system is based on an encoder-decoder recurrent neural network, where the encoder learns a separable feature representation within its hidden states formed by training the model to perform prediction task. We show that according to such unsupervised training the decoder and the encoder self-organize their hidden states into a feature space which clusters similar movements into the same cluster and distinct movements into distant clusters. Current state-of-the-art methods for action recognition are strongly supervised, i.e., rely on providing labels for training. Unsupervised methods have been proposed, however, they require camera and depth inputs (RGB+D) at each time step. In contrast, our system is fully unsupervised, does not require labels of actions at any stage, and can operate with body keypoints input only. Furthermore, the method can perform on various dimensions of body keypoints (2D or 3D) and include additional cues describing movements. We evaluate our system on three extensive action recognition benchmarks with different number of actions and examples. Our results outperform prior unsupervised skeleton-based methods, unsupervised RGB+D based methods on cross-view tests and while being unsupervised have similar performance to supervised skeleton-based action recognition.


While these results are not state-of-the-art (100% reported by #REFR ), this test proves that our model can cope with a larger dataset.

To remove this limitation, we perform a second set of tests where we choose for each class an optimized template stimulus (indicated by Optimized).

This stimulus is determined by iterating through the training dataset and retaining the class-specific training picture that results in the highest training accuracy after a single training epoch.

Table 4 shows the accuracy of the classifications for the different training schemes.

Using the first images in the dataset for each class (condition First), we obtain a testing accuracy of 73%.

This was in the range of the accuracies for training with individual avatars, which range from 66% to 79%.

When the model have access to every domain (MD-NRE-I), the test accuracy reaches a value of 92.15%, which exceeds the accuracy (89.02%) reported in the original paper on the FERG dataset #OTHEREFR .

Moreover, the best transfer learning model (MD-NRE-Bonnie) reaches 84.42% accuracy.

This is an encouraging transfer learning result for a classifier trained with only 12 images, and where all the expressions are from a single avatar type.

Multi-Domain Norm-Referenced Encoding Enables Data Efficient Transfer Learning Of Facial Expression Recognition *

Abstract: People can innately recognize human facial expressions in unnatural forms, such as when depicted on the unusual faces drawn in cartoons or when applied to an animal's features. However, current machine learning algorithms struggle with out-of-domain transfer in facial expression recognition (FER). We propose a biologically-inspired mechanism for such transfer learning, which is based on norm-referenced encoding, where patterns are encoded in terms of difference vectors relative to a domain-specific reference vector. By incorporating domain-specific reference frames, we demonstrate high data efficiency in transfer learning across multiple domains. Our proposed architecture provides an explanation for how the human brain might innately recognize facial expressions on varying head shapes (humans, monkeys, and cartoon avatars) without extensive training. Norm-referenced encoding also allows the intensity of the expression to be read out directly from neural unit activity, similar to face-selective neurons in the brain. Our model achieves a classification accuracy of 92.15% on the FERG dataset with extreme data efficiency. We train our proposed mechanism with only 12 images, including a single image of each class (facial expression) and one image per domain (avatar). In comparison, the authors of the FERG dataset achieved a classification accuracy of 89.02% with their FaceExpr model, which was trained on 43,000 images.

Vgan-Based Image Representation Learning For Privacy-Preserving Facial Expression Recognition

Abstract: Reliable facial expression recognition plays a critical role in human-machine interactions. However, most of the facial expression analysis methodologies proposed to date pay little or no attention to the protection of a user's privacy. In this paper, we propose a Privacy-Preserving Representation-Learning Variational Generative Adversarial Network (PPRL-VGAN) to learn an image representation that is explicitly disentangled from the identity information. At the same time, this representation is discriminative from the standpoint of facial expression recognition and generative as it allows expression-equivalent face image synthesis. We evaluate the proposed model on two public datasets under various threat scenarios. Quantitative and qualitative results demonstrate that our approach strikes a balance between the preservation of privacy and data utility. We further demonstrate that our model can be effectively applied to other tasks such as expression morphing and image completion.


It seems to be the case with other blackbox attacks, since the perturbation is generated randomly using probabilistic laws that don't take into account the fact that document images are brighter than other images #REFR .

Compressing and then decompressing input images of a model using JPEG protocol improves the model robustness against adversarial images, but not consistently.

On the other hand, the adversarial training of both models using the method of Kurakin et al. #OTHEREFR , strongly improves robustness of both models.

This training method is quite easy to implement and does not affect a lot the test accuracy of models on our classification task.

Therefore, this method seems far more effective on a document classification task than against an image classification task, as we can see in Dong et al. #OTHEREFR .

The black-box attack we evaluated generates blurry examples that appear darker than legitimate document images.

There are many ways to improve the robustness of a model that would only use the visual modality of a document #OTHEREFR .

However, state-of-the-art approaches to document classification take advantage of other information modalities, such as the layout of the document, and the text it contains 3 .

Therefore, after this work on evaluating the robustness of visual models, it would be interesting to evaluate the transferability of the generated examples to a multimodal model such as DocFormer or LayoutLMv2, which use optical character recognition (OCR) and transformer layers.

Furthermore, we could explore the possibility of designing adversarial attacks to which these models are more sensitive, for example by targeting OCR prediction errors #OTHEREFR that affect the textual modality and may also affect the robustness of such models #OTHEREFR .

Dealing with the added modality given by text means that more approaches can be explored attacking only one modality or both.

Evaluating Adversarial Robustness On Document Image Classification

Abstract: Adversarial attacks and defenses have gained increasing interest on computer vision systems in recent years, but as of today, most investigations are limited to natural images. However, many artificial intelligence models actually handle documentary data, which is very different from real world images. Hence, in this work, we try to apply the adversarial attack philosophy on documentary data and to protect models against such attacks. Our methodology is to implement untargeted gradient-based, transfer-based and score-based attacks and evaluate the impact of defenses such as adversarial training, JPEG input compression and grey-scale input transformation on the robustness of ResNet50 and EfficientNetB0 model architectures. To the best of our knowledge, no such work has been conducted by the community in order to study the impact of these attacks on the document image classification task.

Efficient Decision-Based Black-Box Adversarial Attacks On Face Recognition

Abstract: Face recognition has obtained remarkable progress in recent years due to the great improvement of deep convolutional neural networks (CNNs). However, deep CNNs are vulnerable to adversarial examples, which can cause fateful consequences in real-world face recognition applications with security-sensitive purposes. Adversarial attacks are widely studied as they can identify the vulnerability of the models before they are deployed. In this paper, we evaluate the robustness of state-of-the-art face recognition models in the decision-based black-box attack setting, where the attackers have no access to the model parameters and gradients, but can only acquire hard-label predictions by sending queries to the target model. This attack setting is more practical in real-world face recognition systems. To improve the efficiency of previous methods, we propose an evolutionary attack algorithm, which can model the local geometry of the search directions and reduce the dimension of the search space. Extensive experiments demonstrate the effectiveness of the proposed method that induces a minimum perturbation to an input face image with fewer queries. We also apply the proposed method to attack a real-world face recognition system successfully.


Depth from Single Image Convolutional networks within an encoder-decoder paradigm #REFR is the standard prototype architecture for dense depth prediction from a single image.

Synthetic dataset generation from computer games #OTHEREFR has been used extensively for training computer vision algorithms like semantic segmentation #OTHEREFR , object detection #OTHEREFR , and depth estimation #OTHEREFR . Mohammad et al.

#OTHEREFR have also used the GTA-V game to generate a synthetic RGB-D dataset but with relative depth at the focus.

The resolution of the dataset used for training is 256×256 which is much smaller than the publicly available datasets.

Furthermore, they need a preprocessing phase, such as histogram equalization, to use the datasets before feeding them to the training.

For training, they use a Resnet architecture and process the RGB image and GT depth with resolution 256 × 256.

The building blocks of such a network consist of convolutional and sub-sampling as their core elements.

However, CNN as an encoder suffers from a local receptive field problem #OTHEREFR , leading to less global representation learning at higher resolutions.

Several algorithms adapt different techniques to learn features at different resolutions to address this issue like dilated convolutions #OTHEREFR or parallel multi-scale feature aggression #OTHEREFR .

Recently, transformer architectures such as vision transformer (ViT) #OTHEREFR or data-efficient image transformers (DeiT) #OTHEREFR have outperformed CNN architectures in image recognition #OTHEREFR , object detection #OTHEREFR , and semantic segmentation #OTHEREFR .

Inspired by the success of transformers in various topics, René et al.

High-Resolution Synthetic Rgb-D Datasets For Monocular Depth Estimation

Abstract: Accurate depth maps are essential in various applications, such as autonomous driving, scene reconstruction, point-cloud creation, etc. However, monocular-depth estimation (MDE) algorithms often fail to provide enough texture & sharpness, and also are inconsistent for homogeneous scenes. These algorithms mostly use CNN or vision transformer-based architectures requiring large datasets for supervised training. But, MDE algorithms trained on available depth datasets do not generalize well and hence fail to perform accurately in diverse real-world scenes. Moreover, the ground-truth depth maps are either lower resolution or sparse leading to relatively inconsistent depth maps. In general, acquiring a high-resolution ground truth dataset with pixel-level precision for accurate depth prediction is an expensive, and time-consuming challenge. In this paper, we generate a high-resolution synthetic depth dataset (HRSD) of dimension 1920 × 1080 from Grand Theft Auto (GTA-V), which contains 100,000 color images and corresponding dense ground truth depth maps. The generated datasets are diverse and have scenes from indoors to outdoors, from homogeneous surfaces to textures. For experiments and analysis, we train the DPT algorithm, a state-of-the-art transformer-based MDE algorithm on the proposed synthetic dataset, which significantly increases the accuracy of depth maps on different scenes by 9%. Since the synthetic datasets are of higher resolution, we propose adding a feature extraction module in the transformer's encoder and incorporating an attention-based loss, further improving the accuracy by 15 %.

High Quality Monocular Depth Estimation Via Transfer Learning

Abstract: Accurate depth estimation from images is a fundamental task in many applications including scene understanding and reconstruction. Existing solutions for depth estimation often produce blurry approximations of low resolution. This paper presents a convolutional neural network for computing a high-resolution depth map given a single RGB image with the help of transfer learning. Following a standard encoder-decoder architecture, we leverage features extracted using high performing pre-trained networks when initializing our encoder along with augmentation and training strategies that lead to more accurate results. We show how, even for a very simple decoder, our method is able to achieve detailed high-resolution depth maps. Our network, with fewer parameters and training iterations, outperforms state-of-the-art on two datasets and also produces qualitatively better results that capture object boundaries more faithfully. Code and corresponding pre-trained weights are made publicly available.


As shown in Table A5 , SCADE trained with the same outof-domain prior that we used for the other datasets (which was trained on Taskonomy #REFR ) outperforms the baselines on the Tanks and Temples dataset as well. Moreover, Figure A11 shows qualitative results.

We conduct further experiments to test the robustness of SCADE.

We evaluate on three scenes from the Tanks and Temples #OTHEREFR dataset, namely three large indoor rooms -Church, Courtroom and Auditorium scenes.

The training set consists of 21, 26 and 21 sparse views for the Church, Courtroom and Auditorium scenes respectively, and the test set consists of 8 sparse views, so the amount of data is similar to that used in prior work #OTHEREFR .

We also followed similar data preprocessing steps as prior work #OTHEREFR and ran SfM #OTHEREFR on all images to obtain camera poses for training.

As shown, SCADE is able to recover objects better than the baselines such as the table in the Church, the group of chairs in the Courtroom (second column), and the rows of seats in the Auditorium (clearer in the side-view seats on the second column).

Moreover, results also show that SCADE avoids clouds of dust such as the lights on the wall of the Church (second column), painting on the wall of the Courtroom (last column) and details on the repetitive seats of the auditorium.

Scade: Nerfs From Space Carving With Ambiguity-Aware Depth Estimates

Abstract: Figure 1. SCADE Overview. We present SCADE, a novel technique for NeRF reconstruction under sparse, unconstrained views for in-the-wild indoor scenes. We leverage on generalizable monocular depth priors and address to represent the inherent ambiguities of monocular depth by exploiting our ambiguity-aware depth estimates (left). Our approach accounts for multimodality of both distributions using our novel space carving loss that seeks to disambiguate and find the common mode to fuse the information between different views (middle). SCADE enables better photometric reconstruction especially in highly ambiguous scenes, e.g. non-opaque surfaces (right).

Taskonomy: Disentangling Task Transfer Learning

Abstract: Do visual tasks have a relationship, or are they unrelated? For instance, could having surface normals simplify estimating the depth of an image? Intuition answers these questions positively, implying existence of a structure among visual tasks. Knowing this structure has notable values; it is the concept underlying transfer learning and provides a principled way for identifying redundancies across tasks, e.g., to seamlessly reuse supervision among related tasks or solve many tasks in one system without piling up the complexity. We proposes a fully computational approach for modeling the structure of space of visual tasks. This is done via finding (first and higher-order) transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space. The product is a computational taxonomic map for task transfer learning. We study the consequences of this structure, e.g. nontrivial emerged relationships, and exploit them to reduce the demand for labeled data. For example, we show that the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3 (compared to training independently) while keeping the performance nearly the same. We provide a set of tools for computing and probing this taxonomical structure including a solver that users can employ to devise efficient supervision policies for their use cases.


Application: Improved 3Sum In Preprocessed Universes

Kopelowitz, Pettie and Porat #REFR gave a simple randomized reduction from 3SUM to O(log n) instances of 3SUM-Convolution via hashing.

As one immediate application, we can solve 3SUM with preprocessed universe, improving Chan and Lewenstein's previous solution which required O(n 13/7 ) query time #OTHEREFR , and also improving Corollary 5.3 regardless of the value of ω: the query algorithm does not use fast matrix multiplication but uses FFT instead, though randomization is now needed in the preprocessing algorithm.

We can preprocess sets A, B, and C of n integers in O(n 2 ) Las Vegas randomized time, so that given any subsets A ′ ⊆ A, B ′ ⊆ B, and C ′ ⊆ C, we can solve All-Nums-3SUM on (A ′ , B ′ , C ′ ) in O(n 11/6 ) time.

The same approach works in the preprocessed universe setting, and transform the input into O(log n) instances where A, B, and C are indexed sets.

During preprocessing, we apply Theorem 10.3 to A ∪ (−B), producing subsets A (λ) and a set R of pairs.

During a query with given subsets A ′ ⊆ A, B ′ ⊆ B, and C ′ ⊆ C, we first examine each pair (a, −b) ∈ R and check whether a ∈ A ′ , b ∈ B ′ , and a + b ∈ C ′ . This takes O(n 2 /s) time.

Next, for each λ, we compute (A (λ) ∩ A ′ ) + ((−A (λ) ) ∩ B ′ ) by known FFT-based algorithms #OTHEREFR ; the running time is near-linear in the output size, which is bounded by |A (λ) − A (λ) |.

For each output value c, we check whether c ∈ C ′ .

Fredman'S Trick Meets Dominance Product: Fine-Grained Complexity Of Unweighted Apsp, 3Sum Counting, And More

Abstract: In this paper we carefully combine Fredman's trick [SICOMP'76] and Matoušek's approach for dominance product [IPL'91] to obtain powerful results in fine-grained complexity: • Under the hypothesis that APSP for undirected graphs with edge weights in {1, 2,. .. , n} requires n 3−o(1) time (when ω = 2), we show a variety of conditional lower bounds, including an n 7/3−o(1) lower bound for unweighted directed APSP and an n 2.2−o(1) lower bound for computing the Minimum Witness Product between two n × n Boolean matrices, even if ω = 2, improving upon their trivial n 2 lower bounds. Our techniques can also be used to reduce the unweighted directed APSP problem to other problems. In particular, we show that (when ω = 2), if unweighted directed APSP requires n 2.5−o(1) time, then Minimum Witness Product requires n 7/3−o(1) time.

Higher Lower Bounds From The 3Sum Conjecture

Abstract: The 3SUM conjecture has proven to be a valuable tool for proving conditional lower bounds on dynamic data structures and graph problems. This line of work was initiated by P\v{a}tra\c{s}cu (STOC 2010) who reduced 3SUM to an offline SetDisjointness problem. However, the reduction introduced by P\v{a}tra\c{s}cu suffers from several inefficiencies, making it difficult to obtain tight conditional lower bounds from the 3SUM conjecture. ::: In this paper we address many of the deficiencies of P\v{a}tra\c{s}cu's framework. We give new and efficient reductions from 3SUM to offline SetDisjointness and offline SetIntersection (the reporting version of SetDisjointness) which leads to polynomially higher lower bounds on several problems. Using our reductions, we are able to show the essential optimality of several algorithms, assuming the 3SUM conjecture. ::: - Chiba and Nishizeki's $O(m\alpha)$-time algorithm (SICOMP 1985) for enumerating all triangles in a graph with arboricity/degeneracy $\alpha$ is essentially optimal, for any $\alpha$. ::: - Bj{\o}rklund, Pagh, Williams, and Zwick's algorithm (ICALP 2014) for listing $t$ triangles is essentially optimal (assuming the matrix multiplication exponent is $\omega=2$). ::: - Any static data structure for SetDisjointness that answers queries in constant time must spend $\Omega(N^{2-o(1)})$ time in preprocessing, where $N$ is the size of the set system. ::: These statements were unattainable via P\v{a}tra\c{s}cu's reductions. ::: We also introduce several new reductions from 3SUM to pattern matching problems and dynamic graph problems. Of particular interest are new conditional lower bounds for dynamic versions of Maximum Cardinality Matching, which introduce a new technique for obtaining amortized lower bounds.


Results On Single-Coil Mri Reconstruction

It is worth noting that even compared to the second best model DCCNN #REFR , PSNR is improved by 0.41dB. This effectively illustrates the excellence of EAMRI.

In Table 1 , we show the quantitative results for all model reconstructions on the Calgary single-coil brain dataset.

According to the table, we can observe that EAMRI achieves the best results for the quantification metrics under acceleration factor 4 and requires fewer model parameters (123K).

6 , we provide a visual comparison of the reconstruction results of these models.

We can see that EAMRI has fewer bright spots in the heatmaps, which means less error between the EAMRI reconstructed image and the ground truth image.

Meanwhile, according to the zoomed-in images of the selected areas, we can observe that our EAMRI can reconstruct more clean and accurate edges. This further validates the validity of EAMRI.

Both the quantitative and the qualitative results for the single-coil MRI reconstruction demonstrate the effectiveness of EAMRI.

Fast Mri Reconstruction Via Edge Attention

Abstract: Fast and accurate MRI reconstruction is a key concern in modern clinical practice. Recently, numerous Deep-Learning methods have been proposed for MRI reconstruction, however, they usually fail to reconstruct sharp details from the subsampled k-space data. To solve this problem, we propose a lightweight and accurate Edge Attention MRI Reconstruction Network (EAMRI) to reconstruct images with edge guidance. Specifically, we design an efficient Edge Prediction Network to directly predict accurate edges from the blurred image. Meanwhile, we propose a novel Edge Attention Module (EAM) to guide the image reconstruction utilizing the extracted edge priors, as inspired by the popular self-attention mechanism. EAM first projects the input image and edges into Q image , K edge , and V image , respectively. Then EAM pairs the Q image with K edge along the channel dimension, such that 1) it can search globally for the high-frequency image features that are activated by the edge priors; 2) the overall computation burdens are largely reduced compared with the traditional spatial-wise attention. With the help of EAM, the predicted edge priors can effectively guide the model to reconstruct high-quality MR images with accurate edges. Extensive experiments show that our proposed EAMRI outperforms other methods with fewer parameters and can recover more accurate edges.

A Deep Cascade Of Convolutional Neural Networks For Dynamic Mr Image Reconstruction

Abstract: Inspired by recent advances in deep learning, we propose a framework for reconstructing dynamic sequences of 2-D cardiac magnetic resonance (MR) images from undersampled data using a deep cascade of convolutional neural networks (CNNs) to accelerate the data acquisition process. In particular, we address the case where data are acquired using aggressive Cartesian undersampling. First, we show that when each 2-D image frame is reconstructed independently, the proposed method outperforms state-of-the-art 2-D compressed sensing approaches, such as dictionary learning-based MR image reconstruction, in terms of reconstruction error and reconstruction speed. Second, when reconstructing the frames of the sequences jointly, we demonstrate that CNNs can learn spatio-temporal correlations efficiently by combining convolution and data sharing approaches. We show that the proposed method consistently outperforms state-of-the-art methods and is capable of preserving anatomical structure more faithfully up to 11-fold undersampling. Moreover, reconstruction is very fast: each complete dynamic sequence can be reconstructed in less than 10 s and, for the 2-D case, each image frame can be reconstructed in 23 ms, enabling real-time applications.


Self-Supervised Action Representation Learning

To learn the contextual coherence in action representation, Lai and Xie #REFR matched pixelwise correspondence from the spatial-temporal color information. Han et al.

#OTHEREFR first proposed a convolutional auto-encoder to construct the latent space from the encoder output.

By operating the high-level features, the model is functional in many areas, such as action interpolation and comparison.

Later on, with a hierarchical RNN auto-encoder in both spatial and temporal domains, Wang et al.

#OTHEREFR developed a high-quality representation space that is motivated for precise action modeling.

As a vision-based learning task, the effectiveness of self-supervised representation is frequently explored in RGB-based action understanding.

#OTHEREFR exploited action representations from multiple modalities of RGB streams and optical flow.

However, these RGB-based action recognition models usually learn contrastive representation from background or visual consistency #OTHEREFR .

With the visual information unavailable, the challenge of learning self-supervised skeletonbased action representation mainly comes from the diverse pose information under different view observations #OTHEREFR .

In self-supervised skeleton-based action representation learning, existing works such as #OTHEREFR mainly focus on preserving the action-dependent features as much as possible to identify samples #OTHEREFR .

For example, an adversarial discriminator is used to assist the auto-encoder to rectify the reconstructed action for a more distinctive representation #OTHEREFR .

Focalized Contrastive View-Invariant Learning For Self-Supervised Skeleton-Based Action Recognition

Abstract: Learning view-invariant representation is a key to improving feature discrimination power for skeleton-based action recognition. Existing approaches cannot effectively remove the impact of viewpoint due to the implicit view-dependent representations. In this work, we propose a self-supervised framework called Focalized Contrastive View-invariant Learning (FoCoViL), which significantly suppresses the view-specific information on the representation space where the viewpoints are coarsely aligned. By maximizing mutual information with an effective contrastive loss between multi-view sample pairs, FoCoViL associates actions with common view-invariant properties and simultaneously separates the dissimilar ones. We further propose an adaptive focalization method based on pairwise similarity to enhance contrastive learning for a clearer cluster boundary in the learned space. Different from many existing self-supervised representation learning work that rely heavily on supervised classifiers, FoCoViL performs well on both unsupervised and supervised classifiers with superior recognition performance. Extensive experiments also show that the proposed contrastive-based focalization generates a more discriminative latent representation.

Self-Supervised Learning For Video Correspondence Flow

Abstract: The objective of this paper is self-supervised learning of feature embeddings that are suitable for matching correspondences along the videos, which we term correspondence flow. By leveraging the natural spatial-temporal coherence in videos, we propose to train a ``pointer'' that reconstructs a target frame by copying pixels from a reference frame. ::: We make the following contributions: First, we introduce a simple information bottleneck that forces the model to learn robust features for correspondence matching, and prevent it from learning trivial solutions, \eg matching based on low-level colour information. Second, to tackle the challenges from tracker drifting, due to complex object deformations, illumination changes and occlusions, we propose to train a recursive model over long temporal windows with scheduled sampling and cycle consistency. Third, we achieve state-of-the-art performance on DAVIS 2017 video segmentation and JHMDB keypoint tracking tasks, outperforming all previous self-supervised learning approaches by a significant margin. Fourth, in order to shed light on the potential of self-supervised learning on the task of video correspondence flow, we probe the upper bound by training on additional data, \ie more diverse videos, further demonstrating significant improvements on video segmentation.

action representation
pixelwise correspondence


Intuitively, ensembles and checkpoint averages also reflect the idea of different views and interpretations of the data and, therefore, provide a framework for uncertainty modeling. Stochastic Weight Averaging (SWA, #REFR ) and SWA-Gaussian (SWAG, Maddox et al. (2019)) both build on this idea.

The approach to uncertainty modeling that we consider is related to the well-established technique of model ensembling.

Stochastic optimization procedures applied in training deep neural networks are non-deterministic and depend on hyper-parameters and initial seeds.

Ensembles have been used as a pragmatic solution to average over several solutions, and the positive impact on model performance pushed ensembling into the standard toolbox of deep learning.

Related to ensembling is the technique of checkpoint averaging (refer to e.g.

#OTHEREFR , which is also known to improve performance.

SWA proposes using the first moments of the parameters of the solutions traversed by the optimizer during the optimization process, as mean estimates of the model parameters.

Using such mean values have been argued to result in finding wider optima, providing better generalization to unseen data.

On top of these mean estimations procured by SWA, SWAG then adds a low-rank plus diagonal approximation of covariances, which, when combined with the aforementioned mean estimations, provide us with corresponding Gaussian posterior approximations over model parameters.

Posterior distribution approximations learned this way then represent our epistemic uncertainty about the model #OTHEREFR , meaning the uncertainty stemming from not knowing the perfect values of the model parameters, since we do not have infinite data to train on.

During test time, instead of making estimates from a single model with deterministic parameters, we sample N different models from the approximated posteriors for each model parameter, and use the average of their prediction distributions as the model response.

Uncertainty-Aware Natural Language Inference With Stochastic Weight Averaging

Abstract: This paper introduces Bayesian uncertainty modeling using Stochastic Weight Averaging-Gaussian (SWAG) in Natural Language Understanding (NLU) tasks. We apply the approach to standard tasks in natural language inference (NLI) and demonstrate the effectiveness of the method in terms of prediction accuracy and correlation with human annotation disagreements. We argue that the uncertainty representations in SWAG better reflect subjective interpretation and the natural variation that is also present in human language understanding. The results reveal the importance of uncertainty modeling, an often neglected aspect of neural language modeling, in NLU tasks.

Averaging Weights Leads To Wider Optima And Better Generalization

Abstract: Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) procedure finds much broader optima than SGD, and approximates the recent Fast Geometric Ensembling (FGE) approach with a single model. Using SWA we achieve notable improvement in test accuracy over conventional SGD training on a range of state-of-the-art residual networks, PyramidNets, DenseNets, and Shake-Shake networks on CIFAR-10, CIFAR-100, and ImageNet. In short, SWA is extremely easy to implement, improves generalization, and has almost no computational overhead.

uncertainty modeling
Stochastic Weight Averaging


We evaluate all methods on the dataset provided in the challenge of Real-World Super-Resolution: NTIRE2020 Track1 and Tracks #REFR .

Specifically, we adopt the same loss functions of Real-ESRGAN #OTHEREFR , which further introduce perceptual loss and adversarial loss to the basic L 1 loss.

We set the learning rate of the KDSR T to 2 × 10 −4 .

We further validate the effectiveness of KDSR on Real-World datasets.

For optimization, we use Adam with β 1 = 0.9, β 2 = 0.99.

In both two stages of training, we set the batch size to 64, with the input patch size being 64.

In addition, we also validate our DiffIR on RealSRSet #OTHEREFR .

Since NTIRE2020 Track1 and RealSRSet datasets provide a paired validation set, we use the LPIPS #OTHEREFR , DISTS #OTHEREFR , and PSNR for the evaluation.

The quantitative results are shown in Tab. 6.

We can see that DiffIR S2 outperforms SOTA real-world SR method KDSR S -GAN on LPIPS, DISTS, and PSNR, consuming fewer computational costs.

In addition, we can see that DiffIR S2 outperforms classic real-world SR method Real-ESRGAN on LPIPS, DISTS, and PSNR, only consuming its 63% Mult-Adds.

Diffir: Efficient Diffusion Model For Image Restoration

Abstract: Diffusion model (DM) has achieved SOTA performance by modeling the image synthesis process into a sequential application of a denoising network. However, different from image synthesis generating each pixel from scratch, most pixels of image restoration (IR) are given. Thus, for IR, traditional DMs running massive iterations on a large model to estimate whole images or feature maps is inefficient. To address this issue, we propose an efficient DM for IR (Dif-fIR), which consists of a compact IR prior extraction network (CPEN), dynamic IR transformer (DIRformer), and denoising network. Specifically, DiffIR has two training stages: pretraining and training DM. In pretraining, we input ground-truth images into CPEN S1 to capture a compact IR prior representation (IPR) to guide DIRformer. In the second stage, we train the DM to directly estimate the same IRP as pretrained CPEN S1 only using LQ images. We observe that since the IPR is only a compact vector, DiffIR can use fewer iterations than traditional DM to obtain accurate estimations and generate more stable and realistic results. Since the iterations are few, our DiffIR can adopt a joint optimization of CPEN S2 , DIRformer, and denoising network, which can further reduce the estimation error influence. We conduct extensive experiments on several IR tasks and achieve SOTA performance while consuming less computational costs.

Aim 2019 Challenge On Real-World Image Super-Resolution: Methods And Results

Abstract: This paper reviews the AIM 2019 challenge on real world super-resolution. It focuses on the participating methods and final results. The challenge addresses the real world setting, where paired true high and low-resolution images are unavailable. For training, only one set of source input images is therefore provided in the challenge. In Track 1: Source Domain the aim is to super-resolve such images while preserving the low level image characteristics of the source input domain. In Track 2: Target Domain a set of high-quality images is also provided for training, that defines the output domain and desired quality of the super-resolved images. To allow for quantitative evaluation, the source input images in both tracks are constructed using artificial, but realistic, image degradations. The challenge is the first of its kind, aiming to advance the state-of-the-art and provide a standard benchmark for this newly emerging task. In total 7 teams competed in the final testing phase, demonstrating new and innovative solutions to the problem.


HGT #REFR uses node-type and edge-type dependent parameters to characterize the heterogeneous attention over each edge based on metarelation, which achieves the state-of-the-art in the mission of knowledge embedding.

Knowledge graph embedding aims to project entities and relations in the knowledge graph into a low-dimensional space.

Graph Neural Networks (GNNs), such as GCN #OTHEREFR and GAT #OTHEREFR , use information aggregation from nearby nodes to learn graph representations. Wang et al.

#OTHEREFR proposed AM-GCN to improve the embedding capability of GCN.

This is because GCN may struggle to learn complex correlation information between node features and topological structures. However, in real-world applications, knowledge graphs can be heterogeneous.

RGCN #OTHEREFR is used as the graph embedding model in MedRec #OTHEREFR which achieve the state-of-theart performance in drug recommendation.

Current studies on knowledge-based recommendation suffer a lot from the problem of graph sparsity, which severely reduces the effectiveness of identifying valuable items for users.

Meanwhile, heterogeneous graph neural network becomes popular for its capacity to incorporate both structured and unstructured information.

But the use of heterogeneous network in recommendation systems has not been extensively investigated.

Enhancing Clinical Evidence Recommendation With Multi-Channel Heterogeneous Learning On Evidence Graphs

Abstract: Clinical evidence encompasses the associations and impacts between patients, interventions (such as drugs or physiotherapy), problems, and outcomes. The goal of recommending clinical evidence is to provide medical practitioners with relevant information to support their decision-making processes and to generate new evidence. Our specific task focuses on recommending evidence based on clinical problems. However, the direct connections between certain clinical problems and related evidence are often sparse, creating a challenge of link sparsity. Additionally, to recommend appropriate evidence, it is essential to jointly exploit both topological relationships among evidence and textual information describing them. To address these challenges, we define two knowledge graphs: an Evidence Co-reference Graph and an Evidence Text Graph, to represent the topological and linguistic relations among evidential elements, respectively. We also introduce a multi-channel heterogeneous learning model and a fusional attention mechanism to handle the coreference-text heterogeneity in evidence recommendation. Our experiments demonstrate that our model outperforms state-of-the-art methods on open data.

Heterogeneous Graph Transformer

Abstract: Recent years have witnessed the emerging success of graph neural networks (GNNs) for modeling structured data. However, most GNNs are designed for homogeneous graphs, in which all nodes and edges belong to the same types, making them infeasible to represent heterogeneous structures. In this paper, we present the Heterogeneous Graph Transformer (HGT) architecture for modeling Web-scale heterogeneous graphs. To model heterogeneity, we design node- and edge-type dependent parameters to characterize the heterogeneous attention over each edge, empowering HGT to maintain dedicated representations for different types of nodes and edges. To handle dynamic heterogeneous graphs, we introduce the relative temporal encoding technique into HGT, which is able to capture the dynamic structural dependency with arbitrary durations. To handle Web-scale graph data, we design the heterogeneous mini-batch graph sampling algorithm---HGSampling---for efficient and scalable training. Extensive experiments on the Open Academic Graph of 179 million nodes and 2 billion edges show that the proposed HGT model consistently outperforms all the state-of-the-art GNN baselines by 9%--21% on various downstream tasks.


HiPPI #REFR is a computationally efficient method that takes geometric relations into account while generalising permutation synchronization but is still limited in resolution.

When more than two shapes of the same class exist, stronger geometric cues can be leveraged to improve results by matching all of them simultaneously.

Unfortunately, the already very high problem complexity increases even further the more shapes are used.

Hence, existing multi-shape matching methods limit the total number of shapes and their resolution #OTHEREFR , work in spectral space #OTHEREFR , or relax the permutation constraints #OTHEREFR .

Early multi-matching methods computed pair-wise matchings and subsequently used permutation synchronisation to establish cycle consistency #OTHEREFR .

Still, permutation synchronisation requires the eigendecomposition of a matrix with quadratically increasing dimensions #OTHEREFR .

Instead of looking at permutations directly, ZoomOut #OTHEREFR reduces the dimensionality of the problem by projecting it onto the spectral decomposition.

This idea has been extended to take cycle consistency within the spectral space into account #OTHEREFR , which does not guarantee a point-wise consistent matching.

To circumvent this issue, IsoMuSh #OTHEREFR jointly optimises point and functional correspondences.

The method detangles the optimisation into smaller subproblems by using a so-called universe shape that all shapes are mapped to instead of each other, as Cao and Bernard do #OTHEREFR .

Using a universe is similar to requiring a template shape, as many learning-based approaches do #OTHEREFR : Both synchronise all correspondences by matching them through a unified space.

Ccuantumm: Cycle-Consistent Quantum-Hybrid Matching Of Multiple Shapes

Abstract: Jointly matching multiple, non-rigidly deformed 3D shapes is a challenging, N P-hard problem. A perfect matching is necessarily cycle-consistent: Following the pairwise point correspondences along several shapes must end up at the starting vertex of the original shape. Unfortunately, existing quantum shape-matching methods do not support multiple shapes and even less cycle consistency. This paper addresses the open challenges and introduces the first quantum-hybrid approach for 3D shape multimatching; in addition, it is also cycle-consistent. Its iterative formulation is admissible to modern adiabatic quantum hardware and scales linearly with the total number of input shapes. Both these characteristics are achieved by reducing the N-shape case to a sequence of three-shape matchings, the derivation of which is our main technical contribution. Thanks to quantum annealing, high-quality solutions with low energy are retrieved for the intermediate N Phard objectives. On benchmark datasets, the proposed approach significantly outperforms extensions to multi-shape matching of a previous quantum-hybrid two-shape matching method and is on-par with classical multi-matching methods. Our source code is available at 4dqv.mpiinf.mpg.de/CCuantuMM/.

Hippi: Higher-Order Projected Power Iterations For Scalable Multi-Matching

Abstract: The matching of multiple objects (e.g. shapes or images) is a fundamental problem in vision and graphics. In order to robustly handle ambiguities, noise and repetitive patterns in challenging real-world settings, it is essential to take geometric consistency between points into account. Computationally, the multi-matching problem is difficult. It can be phrased as simultaneously solving multiple (NP-hard) quadratic assignment problems (QAPs) that are coupled via cycle-consistency constraints. The main limitations of existing multi-matching methods are that they either ignore geometric consistency and thus have limited robustness, or they are restricted to small-scale problems due to their (relatively) high computational cost. We address these shortcomings by introducing a Higher-order Projected Power Iteration method, which is (i) efficient and scales to tens of thousands of points, (ii) straightforward to implement, (iii) able to incorporate geometric consistency, (iv) guarantees cycle-consistent multi-matchings, and (iv) comes with theoretical convergence guarantees. Experimentally we show that our approach is superior to existing methods.


Simulation Of The Stable Undershoot Via Domination By A Mixture Of Densities

In the following two subsections we describe how to sample from the laws given by the densities ψ The bivariate density ψ #REFR s can be written as the product of a marginal log-concave density and an exponential density with deterministic shift and a random scale.

The random running time of Algorithm 5 has exponential moments.

Moreover, the expected running time of Algorithm 5 is bounded above by

where the constant κ 5 > 0 depends on neither s = (θt) −1/α w ∈ (0, ∞) (see line 1 of Algorithm 5), t > 0, θ ∈ (0, ∞) nor α ∈ (0, 1).

The proof of Proposition 2.5 is given in Subsection 4.3.4 below.

The latter is easy to simulate and the former can also be simulated via the general Algorithm 11 by virtue of being log-concave.

Require: Parameters α ∈ (0, 1) and s > 0

1: Sample Y s with density y → exp(−σ α (y)s −r ) via Algorithm 11 2: Sample E ∼ Exp(1) and return (s −r + E/σ α (Y s ), Y s )

Algorithm 6 samples from the density ψ #OTHEREFR s .

Fast Exact Simulation Of The First Passage Of A Tempered Stable Subordinator Across A Non-Increasing Function

Abstract: We construct a fast exact algorithm for the simulation of the first-passage time, jointly with the undershoot and overshoot, of a tempered stable subordinator over an arbitrary non-increasing absolutely continuous function. We prove that the running time of our algorithm has finite exponential moments and provide bounds on its expected running time with explicit dependence on the characteristics of the process and the initial value of the function. The expected running time grows at most cubically in the stability parameter (as it approaches either 0 or 1) and is linear in the tempering parameter and the initial value of the function. Numerical performance, based on the implementation in the dedicated GitHub repository, exhibits a good agreement with our theoretical bounds. We provide numerical examples to illustrate the performance of our algorithm in Monte Carlo estimation.

Accurate And Efficient Numerical Calculation Of Stable Densities Via Optimized Quadrature And Asymptotics

Abstract: Stable distributions are an important class of infinitely-divisible probability distributions, of which two special cases are the Cauchy distribution and the normal distribution. Aside from a few special cases, the density function for stable distributions has no known analytic form, and is expressible only through the variate's characteristic function or other integral forms. In this paper we present numerical schemes for evaluating the density function for stable distributions, its gradient, and distribution function in various parameter regimes of interest, some of which had no pre-existing efficient method for their computation. The novel evaluation schemes consist of optimized generalized Gaussian quadrature rules for integral representations of the density function, complemented by various asymptotic expansions near various values of the shape and argument parameters. We report several numerical examples illustrating the efficiency of our methods. The resulting code has been made available online.


And numerical results are surprisingly good, i.e., 27.2 AP with SID #REFR , which outperforms baseline by 7.4 points. This implies the superiority of using RAW images.

And ResNet-50-FPN #OTHEREFR Faster R-CNN 22.1 35.6 23.5 GLADNet #OTHEREFR Faster R-CNN 15.4 24.9 16.4 Retinex-Net #OTHEREFR Faster R-CNN 19.6 31.1 21.5 EnlightenGAN #OTHEREFR Faster R-CNN 21.1 34.8 21.9 Zero-DCE #OTHEREFR Faster R-CNN 22.0 35.9 23.5

Enhance + Denoise HE #OTHEREFR ) + SGN #OTHEREFR Faster R-CNN 25.1 39.8 26.9 GLADNet #OTHEREFR ) + SGN #OTHEREFR Faster R-CNN 24.1 39.1 25.3 Retinex-Net #OTHEREFR ) + SGN #OTHEREFR Faster R-CNN 25.5 41.0 27.4 EnlightenGAN #OTHEREFR itively expect performance improvement, but the accuracies stay the same (with histogram equalization #OTHEREFR and Zero-DCE #OTHEREFR ) or even decrease (with GLADNet #OTHEREFR , Retinex-Net #OTHEREFR and EnlightenGAN #OTHEREFR ).

We guess the reason is that these enhancers only improve the overall brightness but cannot handle the noise.

To verify it, we further introduce denoiser to the pipeline, and the overall accuracy significantly increases as expected, e.g., Zero-DCE #OTHEREFR plus SGN #OTHEREFR leads to 6.7 AP gain. Notice that these methods for comparison use camera outputs.

Then, we also perform experiments with SID #OTHEREFR and REDI #OTHEREFR , which can restore sRGB images from low-light RAW images.

Though these enhancing and denoising steps boost the low-light instance segmentation performance remarkably, our method achieves the best quantitative results without extra preprocessing steps.

Besides, the inference speed of the pro-posed method outperforms all other pipelines.

And its speed is very close to the original Mask R-CNN #OTHEREFR .

Moreover, qualitative results illustrated in Figure 12 show the proposed method can consistently recall most of the targets even in challenging scenarios.

Instance Segmentation In The Dark

Abstract: Existing instance segmentation techniques are primarily tailored for high-visibility inputs, but their performance significantly deteriorates in extremely low-light environments. In this work, we take a deep look at instance segmentation in the dark and introduce several techniques that substantially boost the low-light inference accuracy. The proposed method is motivated by the observation that noise in low-light images introduces high-frequency disturbances to the feature maps of neural networks, thereby significantly degrading performance. To suppress this "feature noise", we propose a novel learning method that relies on an adaptive weighted downsampling layer, a smooth-oriented convolutional block, and disturbance suppression learning. These components effectively reduce feature noise during downsampling and convolution operations, enabling the model to learn disturbance-invariant features. Furthermore, we discover that high-bit-depth RAW images can better preserve richer scene information in low-light conditions compared to typical camera sRGB outputs, thus supporting the use of RAW-input algorithms. Our analysis indicates that high bit

Learning To See In The Dark

Abstract: Imaging in low light is challenging due to low photon count and low SNR. Short-exposure images suffer from noise, while long exposure can induce blur and is often impractical. A variety of denoising, deblurring, and enhancement techniques have been proposed, but their effectiveness is limited in extreme conditions, such as video-rate imaging at night. To support the development of learning-based pipelines for low-light image processing, we introduce a dataset of raw short-exposure low-light images, with corresponding long-exposure reference images. Using the presented dataset, we develop a pipeline for processing low-light images, based on end-to-end training of a fully-convolutional network. The network operates directly on raw sensor data and replaces much of the traditional image processing pipeline, which tends to perform poorly on such data. We report promising results on the new dataset, analyze factors that affect performance, and highlight opportunities for future work. The results are shown in the supplementary video at this https URL


Figure 3 also ends with an LAS decoder, except this one can optionally attend to the continuous encoder features as well, as is done in previous deliberation work #REFR . Gradients do not flow back through embedded N-best.

A fitting baseline to this experiment is second-pass deliberation ASR #OTHEREFR .

Typically, a deliberation system generates first-pass hypotheses using a fast decoder, like RNN-T, then embeds its Nbest hyps and attends to them with a second-pass full-context LAS decoder.

We have therefore constructed a comparable deliberation baseline model shown in Figure 3 . This model is analogous to our full pipeline, i.e.

Figures 1 & 2 put together, and is designed to have a similar total model size and encoder latency.

It starts with the same frozen base encoder, then trains a first-pass RNN-T decoder to obtain the N-best hyps, which stands to be compared to the Lego-Features in terms of informativeness and modularity.

Lego-Features: Exporting Modular Encoder Features For Streaming And Deliberation Asr

Abstract: In end-to-end (E2E) speech recognition models, a representational tight-coupling inevitably emerges between the encoder and the decoder. We build upon recent work that has begun to explore building encoders with modular encoded representations, such that encoders and decoders from different models can be stitched together in a zero-shot manner without further fine-tuning. While previous research only addresses full-context speech models, we explore the problem in a streaming setting as well. Our framework builds on top of existing encoded representations, converting them to modular features, dubbed as Lego-Features, without modifying the pre-trained model. The features remain interchangeable when the model is retrained with distinct initializations. Though sparse, we show that the Lego-Features are powerful when tested with RNN-T or LAS decoders, maintaining high-quality downstream performance. They are also rich enough to represent the first-pass prediction during twopass deliberation. In this scenario, they outperform the N-best hypotheses, since they do not need to be supplemented with acoustic features to deliver the best results. Moreover, generating the Lego-Features does not require beam search or auto-regressive computation. Overall, they present a modular, powerful and cheap alternative to the standard encoder output, as well as the N-best hypotheses.

Deliberation Model Based Two-Pass End-To-End Speech Recognition

Abstract: End-to-end (E2E) models have made rapid progress in automatic speech recognition (ASR) and perform competitively relative to conventional models. To further improve the quality, a two-pass model has been proposed to rescore streamed hypotheses using the non-streaming Listen, Attend and Spell (LAS) model while maintaining a reasonable latency. The model attends to acoustics to rescore hypotheses, as opposed to a class of neural correction models that use only first-pass text hypotheses. In this work, we propose to attend to both acoustics and first-pass hypotheses using a deliberation network. A bidirectional encoder is used to extract context information from first-pass hypotheses. The proposed deliberation model achieves 12% relative WER reduction compared to LAS rescoring in Google Voice Search (VS) tasks, and 23% reduction on a proper noun test set. Compared to a large conventional model, our best model performs 21% relatively better for VS. In terms of computational complexity, the deliberation decoder has a larger size than the LAS decoder, and hence requires more computations in second-pass decoding.


A similar result was derived in #REFR from a probabilistic point of view where the signal support is assumed to be formed by two subsets with different probability of occurrence.

Weights have been employed in sparse recovery methods for various purposes.

For instance, in the seminal work #OTHEREFR , the authors propose to solve a sequence of (re)weighted ℓ 1 minimization problems to enhance sparse signal recovery.

In our context, weights can generally be thought of as a way of incorporating prior information about the signal into a sparse recovery model.

In adaptive LASSO #OTHEREFR , a data-driven but careful choice of weights is shown to admit near oracle properties.

Works such as #OTHEREFR show that replacing the ℓ 1 -norm with its weighted version can improve recovery assuming that accurate (partial) support knowledge is provided.

Further studies of weighted ℓ 1 minimization and its impactful application in the context of function approximation from pointwise samples and uncertainty quantification include #OTHEREFR .

The notion of weighted sparsity was formalized in #OTHEREFR .

Weighted sparsity is related to structured sparsity (see #OTHEREFR ).

In fact it allows one to promote structures (rather than being a structure itself).

For example, in the context of highdimensional function approximation (see #OTHEREFR and references therein) weights are able to promote so-called sparsity in lower sets, which largely contributes to mitigating the curse of dimensionality in the sample complexity.

The Greedy Side Of The Lasso: New Algorithms For Weighted Sparse Recovery Via Loss Function-Based Orthogonal Matching Pursuit

Abstract: We propose a class of greedy algorithms for weighted sparse recovery by considering new loss function-based generalizations of Orthogonal Matching Pursuit (OMP). Given a (regularized) loss function, the proposed algorithms alternate the iterative construction of the signal support via greedy index selection and a signal update based on solving a local data-fitting problem restricted to the current support. We show that greedy selection rules associated with popular weighted sparsity-promoting loss functions admit explicitly computable and simple formulas. Specifically, we consider ℓ 0-and ℓ 1-based versions of the weighted LASSO (Least Absolute Shrinkage and Selection Operator), the Square-Root LASSO (SR-LASSO) and the Least Absolute Deviations LASSO (LAD-LASSO). Through numerical experiments on Gaussian compressive sensing and high-dimensional function approximation, we demonstrate the effectiveness of the proposed algorithms and empirically show that they inherit desirable characteristics from the corresponding loss functions, such as SR-LASSO's noise-blind optimal parameter tuning and LAD-LASSO's fault tolerance. In doing so, our study sheds new light on the connection between greedy sparse recovery and convex relaxation.

Analyzing Weighted $\Ell_1$ Minimization For Sparse Recovery With Nonuniform Sparse Models

Abstract: In this paper we introduce a nonuniform sparsity model and analyze the performance of an optimized weighted $\ell_1$ minimization over that sparsity model. In particular, we focus on a model where the entries of the unknown vector fall into two sets, with entries of each set having a specific probability of being nonzero. We propose a weighted $\ell_1$ minimization recovery algorithm and analyze its performance using a Grassmann angle approach. We compute explicitly the relationship between the system parameters-the weights, the number of measurements, the size of the two sets, the probabilities of being nonzero- so that when i.i.d. random Gaussian measurement matrices are used, the weighted $\ell_1$ minimization recovers a randomly selected signal drawn from the considered sparsity model with overwhelming probability as the problem dimension increases. This allows us to compute the optimal weights. We demonstrate through rigorous analysis and simulations that for the case when the support of the signal can be divided into two different subclasses with unequal sparsity fractions, the optimal weighted $\ell_1$ minimization outperforms the regular $\ell_1$ minimization substantially. We also generalize the results to an arbitrary number of classes.


Additionally, we compared the performance of LOLA-OffPA2 with LOLA-DiCE #REFR , which is designed specifically for this game, and we reported the results in Table 3 .

The LOLA agents can shape the opponent's learning to encourage cooperation and, therefore, converge to TFT #OTHEREFR .

We evaluate the methods' performances based on the Averaged Episode Reward (AER).

Cooperate (−1, −1) (−3, 0) Defect (0, −3) (−3, −3)

In Figure 4 , we depict the learning curves for LOLA-OffPA2 and the MADDPG-based methods.

From this figure, we find that only LOLA-OffPA2 can solve the game, which once again highlights the importance of learning anticipation.

Although both methods demonstrate high values of AER, our LOLA-OffPA2 is significantly more efficient as its LATC value is much lower than that of LOLA-DiCE.

Table 5 : Comparisons of LOLA-DiCE with our proposed LOLA-OffPA2 in the Exit-Room game, in terms of performance (normalized average return in different game levels) and efficiency (learning anticipation time complexity in different reasoning levels).

Off-Policy Action Anticipation In Multi-Agent Reinforcement Learning

Abstract: Learning anticipation in Multi-Agent Reinforcement Learning (MARL) is a reasoning paradigm where agents anticipate the learning steps of other agents to improve cooperation among themselves. As MARL uses gradient-based optimization, learning anticipation requires using Higher-Order Gradients (HOG), with so-called HOG methods. Existing HOG methods are based on policy parameter anticipation, i.e., agents anticipate the changes in policy parameters of other agents. Currently, however, these existing HOG methods have only been applied to differentiable games or games with small state spaces. In this work, we demonstrate that in the case of non-differentiable games with large state spaces, existing HOG methods do not perform well and are inefficient due to their inherent limitations related to policy parameter anticipation and multiple sampling stages. To overcome these problems, we propose Off-Policy Action Anticipation (OffPA2), a novel framework that approaches learning anticipation through action anticipation, i.e., agents anticipate the changes in actions of other agents, via off-policy sampling. We theoretically analyze our proposed OffPA2 and employ it to develop multiple HOG methods that are applicable to non-differentiable games with large state spaces. We conduct a large set of experiments and illustrate that our proposed HOG methods outperform the existing ones regarding efficiency and performance.

Dice: The Infinitely Differentiable Monte-Carlo Estimator

Abstract: The score function estimator is widely used for estimating gradients of stochastic objectives in Stochastic Computation Graphs (SCG), eg. in reinforcement learning and meta-learning. While deriving the first-order gradient estimators by differentiating a surrogate loss (SL) objective is computationally and conceptually simple, using the same approach for higher-order gradients is more challenging. Firstly, analytically deriving and implementing such estimators is laborious and not compliant with automatic differentiation. Secondly, repeatedly applying SL to construct new objectives for each order gradient involves increasingly cumbersome graph manipulations. Lastly, to match the first-order gradient under differentiation, SL treats part of the cost as a fixed sample, which we show leads to missing and wrong terms for higher-order gradient estimators. To address all these shortcomings in a unified way, we introduce DiCE, which provides a single objective that can be differentiated repeatedly, generating correct gradient estimators of any order in SCGs. Unlike SL, DiCE relies on automatic differentiation for performing the requisite graph manipulations. We verify the correctness of DiCE both through a proof and through numerical evaluation of the DiCE gradient estimates. We also use DiCE to propose and evaluate a novel approach for multi-agent learning. Our code is available at this https URL


It is a natural starting point for Hamiltonian simulation since it can be used to formulate many computationally hard problems, such as spin glasses, Quadratic Unconstrained Binary Optimization problems (QUBOs), or graph partitioning #REFR .

This paper will primarily focus on the transverse-field Ising model as a representative of an important class of Hamiltonians.

In its general form, it models the spins of particles and is described by the Hamiltonian

where J ij is the interaction strength between nearest-neighbor spin pairs at sites i, j and g j is an external field pointing perpendicular to the interactions at site j.

The following constrained version of this type of Hamiltonian will be used to illustrate all proposed concepts and methods throughout the remainder of this paper.

Example 1. The L-site finite 1D Ising chain is defined by

where the parameters are site-independent, i.e., J ij := J and g := g.

Using the product formula, a single Trotter under this model has the form

Towards Hamiltonian Simulation With Decision Diagrams

Abstract: This paper proposes a novel approach to Hamiltonian simulation using Decision Diagrams (DDs), which are an exact representation based on exploiting redundancies in representations of quantum states and operations. While the simulation of Hamiltonians has been studied extensively, scaling these simulations to larger or more complex systems is often challenging and may require approximations or new simulation methods altogether. DDs offer such an alternative that has not yet been applied to Hamiltonian simulation. In this work, we investigate the behavior of DDs for this task. To this end, we review the basics of DDs such as their construction and present how the relevant operations for Hamiltonian simulation are implemented in this data structure-leading to the first DD-based Hamiltonian simulation approach. Based on several series of evaluations and comparisons, we then discuss insights about the performance of this complementary approach. Overall, these studies show that DDs indeed may offer a promising new data structure which, for certain examples, can provide orders of magnitudes of improvement compared to the state-of-the-art, yet also comes with its own, fundamentally different, limitations.

Ising Formulations Of Many Np Problems

Abstract: We provide Ising formulations for many NP-complete and NP-hard problems, including all of Karp's 21 NP-complete problems. This collects and extends mappings to the Ising model from partitioning, covering and satisfiability. In each case, the required number of spins is at most cubic in the size of the problem. This work may be useful in designing adiabatic quantum optimization algorithms.

Hamiltonian simulation
many computationally hard


Much more closely aligned to our work, and a natural fit for irregularly sampled data is research that uses differential equations to model continuous-time processes #REFR .

Time series modelling in machine learning: There is vast literature on the use of machine learning for time series modelling and we highlight some of the ideas that have been explored to adapt diverse kinds of models for irregular time series data.

Although not naturally well suited to learning representations of such data, there have been modifications proposed to discrete-time models such as recurrent neural networks #OTHEREFR to handle such data.

Models such as mTANs #OTHEREFR leverage an attention-based approach to interpolate sequences to create discrete-time data from irregularly sampled data.

Another strategy has been architectural modifications to the recurrence equations e.g.

CT-GRU #OTHEREFR , GRU-D #OTHEREFR and Unitary RNNs #OTHEREFR .

By parameterizing the derivative of a time series using neural networks and integrating the dynamics over unobserved time points, this class of models is well suited to handle irregularly sampled data.

This includes models such as ODE-RNN #OTHEREFR , ODE-LSTM #OTHEREFR and Neural CDE #OTHEREFR .

ODE-based approaches require the use of differential equation solvers during training and inference, which can come at the cost of runtime #OTHEREFR .

PolyODEs lie in this family of models; specifically, this work proposes a new parameterization of the dynamics function and a practical method for learning that enables this model family to accurately forecast the future and reconstruct the past greatly enhancing the scope and utility of the learned embeddings.

Orthogonal polynomials: PolyODEs are inspired by a rich line of work in orthogonal decomposition of time series data.

Anamnesic Neural Differential Equations With Orthogonal Polynomials Projections

Abstract: Neural ordinary differential equations (Neural ODEs) are an effective framework for learning dynamical systems from irregularly sampled time series data. These models provide a continuous-time latent representation of the underlying dynamical system where new observations at arbitrary time points can be used to update the latent representation of the dynamical system. Existing parameterizations for the dynamics functions of Neural ODEs limit the ability of the model to retain global information about the time series; specifically, a piece-wise integration of the latent process between observations can result in a loss of memory on the dynamic patterns of previously observed data points. We propose PolyODE, a Neural ODE that models the latent continuous-time process as a projection onto a basis of orthogonal polynomials. This formulation enforces long-range memory and preserves a global representation of the underlying dynamical system. Our construction is backed by favourable theoretical guarantees and in a series of experiments, we demonstrate that it outperforms previous works in the reconstruction of past and future data, and in downstream prediction tasks. Our code is available at https://github.com/edebrouwer/polyode.

Neural Ordinary Differential Equations

Abstract: We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

continuous-time processes
differential equations