If the attack identification is performed with all the measurements, then the number of cases is equal to #REFR 4 = 4845.

, 20, and the noise v(t) ∈ R 20 be given such that v(t) ≤ 0.01.

In this example, the partial information Φ i (x) of the state x obtained from each individual output y i is no more than h i (x), i.e., Φ i = h i , ∀i, as it can be computed from (5) that L f h i = −h i for i ≤ 10, and L f h i = 0 for i ≥ 11.

Then, it can be verified that the state x(t) can be restored from any 12 elements of the measurements {y i (t)} #OTHEREFR i=1 , so that the attack identification and the resilient state estimation is possible when up to 4 measurements are compromised.

But, the attack identification can also be performed with the local measurements {y i (t)} 10 i=1 and {y i (t)} 20 i=11 , separately, although the state x(t) is not observable from either of them.

This is because both the collections {Φ i } 10

Abstract: A resilient state estimation scheme for uniformly observable nonlinear systems, based on a method for local identification of sensor attacks, is presented. The estimation problem is combinatorial in nature, and so many methods require substantial computational and storage resources as the number of sensors increases. To reduce the complexity, the proposed method performs the attack identification with local subsets of the measurements, not with the set of all measurements. A condition for nonlinear attack identification is introduced as a relaxed version of existing redundant observability condition. It is shown that an attack identification can be performed even when the state cannot be recovered from the measurements. As a result, although a portion of measurements are compromised, they can be locally identified and excluded from the state estimation, and thus the true state can be recovered. Simulation results demonstrate the effectiveness of the proposed scheme.

Detection Of Sensor Attack And Resilient State Estimation For Uniformly Observable Nonlinear Systems Having Redundant Sensors

Abstract: This paper presents a detection algorithm for sensor attacks and a resilient state estimation scheme for a class of uniformly observable nonlinear systems. An adversary is supposed to corrupt a subset of sensors with the possibly unbounded signals, while the system has sensor redundancy. We design an individual high-gain observer for each measurement output so that only the observable portion of the system state is obtained. Then, a nonlinear error correcting problem is solved by collecting all the information from those partial observers and exploiting redundancy. A computationally efficient, on-line monitoring scheme is presented for attack detection. Based on the attack detection scheme, an algorithm for resilient state estimation is provided. The simulation results demonstrate the effectiveness of the proposed algorithm.


In #REFR , authors studied the reliability of command and control channel between UAVs and a traditional cellular network with massive MIMO.

A high mobility UAV communication relay was analyzed in #OTHEREFR .

The authors studied the throughput maximization problems by designing UAV trajectory and power allocations.

Authors in #OTHEREFR , #OTHEREFR investigated a Max-Min SNR based UAV trajectory optimization problem.

Authors in #OTHEREFR jointly designed multiple UAVs' trajectories and transmit powers to maximize the throughput of a group of ground users.

In #OTHEREFR , the authors considered a UAV that collects data from multiple wireless sensor networks and jointly optimized UAV's trajectory and communication scheduling to maximize the minimum average data collection rate.

Besides, a deep reinforcement learning-based approach of optimizing multiple cellular-connected UAVs to minimize their interference with ground users was analyzed in #OTHEREFR .

Authors in #OTHEREFR studied the uplink inter-cell interference coordination design for a cellular network simultaneously serving UAVs and ground users.

Typically, the maximum weighted sum-rate of UAVs and users is jointly optimized based on power allocations and uplink cell associations.

Authors in #OTHEREFR maximized the sum rate of UAV uplink transmission by optimizing the precoding vectors at the multi-antenna UAVs.

A deep reinforcement learning-based intelligent navigation task of a cellular-connected UAV network was investigated in #OTHEREFR which aims to minimize the weighted sum of time cost and expected outage duration alongside UAV trajectory.

Coverage Analysis And Trajectory Optimization For Aerial Users With Dedicated Cellular Infrastructure

Abstract: In this paper, we consider a novel cellular network for aerial users, which is composed of dedicated base stations (BSs), whose antennas are directed towards aerial users, and traditional terrestrial BSs (TBSs). Besides, the dedicated BSs are deployed on roadside furniture, such as lampposts and traffic lights, to achieve multiple features while occupying less space. Therefore, the locations of dedicated BSs and TBSs are modeled by a Poisson-line-Cox-process (PLCP) and Poisson point process (PPP), respectively. For the proposed network, we first compute the aerial coverage probability and show that the deployment of dedicated BSs improves the coverage probability in both high dense areas and rural areas. We then consider a cellular-connected UAV that has a flying mission and optimize its trajectory to maximize the minimal achievable signal-to-interference-plus-noise ratio (SINR) (Max-Min SINR). To obtain the Max-Min SINR and minimal time trajectory that satisfies the Max-Min SINR, we proposed two algorithms that are practical in large-scale networks. Finally, our results show that the optimal density of dedicated BSs which maximizes Max-Min SINR decreases with the increase of the road densities.

Understanding Uav Cellular Communications: From Existing Networks To Massive Mimo

Abstract: The purpose of this article is to bestow the reader with a timely study of UAV cellular communications, bridging the gap between the 3GPP standardization status quo and the more forward-looking research. Special emphasis is placed on the downlink command and control (C&C) channel to aerial users, whose reliability is deemed of paramount technological importance for the commercial success of UAV cellular communications. Through a realistic side-by-side comparison of two network deployments -- a present-day cellular infrastructure versus a next-generation massive MIMO system -- a plurality of key facts are cast light upon, with the three main ones summarized as follows: (i) UAV cell selection is essentially driven by the secondary lobes of a base station's radiation pattern, causing UAVs to associate to far-flung cells; (ii) over a 10 MHz bandwidth, and for UAV heights of up to 300 m, massive MIMO networks can support 100 kbps C&C channels in 74% of the cases when the uplink pilots for channel estimation are reused among base station sites, and in 96% of the cases without pilot reuse across the network; (iii) supporting UAV C&C channels can considerably affect the performance of ground users on account of severe pilot contamination, unless suitable power control policies are in place.

traditional cellular network
massive MIMO


Projected gradient descent attack #REFR is a widely-used attack method that applies small steps of maximizing the loss objective iteratively and clipping the values of intermediate results after each step (projection to the norm ball) to ensure that they are in a constrained neighbourhood of the original inputs.

The time series that appear more distinguished to our eyes are captured by larger Wasserstein distances.

The example in the upper figure may not be a successful adversarial example under 1 distance as it is distant in the 1 space. However, it is almost imperceptible to human evaluation.

In this case, Wasserstein distance is a better measurement of adversarial examples.

In this work, we study the adversarial attack on time series in the Wasserstein space for the first time.

Our goal is to generate adversarial examples that have small Wasserstein perturbation so it is more indistinguishable and natural to human, e.g., physician who examines ECG data.

Similarly, we propose a Wasserstein PGD method to search for adversarial examples in the Wasserstein space for univariant time series.

Wasserstein distance cannot be calculated directly without solving an optimization subproblem and has no closed-form solution in most cases, which limits its applications.

At present, there are only two cases that the Wasserstein distance can be directly calculated, one is the case of the dimension of inputs being 1, and the other is the inputs following Gaussian distribution.

For the univariant time series, we can take advantage of its 1 characteristic and use the closedform Wasserstein distance to apply the projection of intermediate results of each step onto the Wasserstein ball with gradient descent method.

The direct projection of intermediate results onto the Wasserstein ball using gradient descent could be time-consuming and converge to a sub-optimal solution.

Wasserstein Adversarial Examples On Univariant Time Series Data

Abstract: Adversarial examples are crafted by adding indistinguishable perturbations to normal examples in order to fool a well-trained deep learning model to misclassify. In the context of computer vision, this notion of indistinguishability is typically bounded by ∞ or other norms. However, these norms are not appropriate for measuring indistinguishiability for time series data. In this work, we propose adversarial examples in the Wasserstein space for time series data for the first time and utilize Wasserstein distance to bound the perturbation between normal examples and adversarial examples. We introduce Wasserstein projected gradient descent (WPGD), an adversarial attack method for perturbing univariant time series data. We leverage the closed-form solution of Wasserstein distance in the 1D space to calculate the projection step of WPGD efficiently with the gradient descent method. We further propose a two-step projection so that the search of adversarial examples in the Wasserstein space is guided and constrained by Euclidean norms to yield more effective and imperceptible perturbations. We empirically evaluate the proposed attack on several time series datasets in the healthcare domain. Extensive results demonstrate that the Wasserstein attack is powerful and can successfully attack most of the target classifiers with a high attack success rate. To better study the nature of Wasserstein adversarial example, we evaluate a strong defense mechanism named Wasserstein smoothing for potential certified robustness defense. Although the defense can achieve some accuracy gain, it still has limitations in many cases and leaves space for developing a stronger certified robustness method to Wasserstein adversarial examples on univariant time series data.

Towards Deep Learning Models Resistant To Adversarial Attacks

Abstract: Recent work has demonstrated that neural networks are vulnerable to adversarial examples, i.e., inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network. In fact, some of the latest findings suggest that the existence of adversarial attacks may be an inherent weakness of deep learning models. To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization. This approach provides us with a broad and unifying view on much of the prior work on this topic. Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal. In particular, they specify a concrete security guarantee that would protect against any adversary. These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks. They also suggest the notion of security against a first-order adversary as a natural and broad security guarantee. We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models.

Projected gradient descent
loss objective


C. Adaptive Hybrid Spatial-Temporal Learning Block

We also apply multi-head attention #REFR to capture the relation semantics between nodes from different learning subspaces.

DGL implements Graph Attention Network (GAT) #OTHEREFR to perform dynamic spatial feature aggregation, owing to GAT's ability to capture the dynamic influences of neighbors.

We believe that short-term dynamic influences mostly occur between nearby cell towers, so we deploy the exponential distance-decay matrix #OTHEREFR as the prior graph structure of GAT.

We call it the distance adjacency matrix, denoted as A dis ∈ R N ×N . As shown in Fig.

2 , GAT achieves weighted feature aggregation by calculating the attention scores of the central node and neighbor nodes.

Moreover, the multi-head attention can be computed in parallel for reducing time complexity.

For simplicity, taking a graph node v i as an example, the graph attention mechanism of DGL can be formulated as follows:

where a ij ∈ R represents the computed attention score between v i and v j , where j ∈ N i .

Adaptive Hybrid Spatial-Temporal Graph Neural Network For Cellular Traffic Prediction

Abstract: Cellular traffic prediction is an indispensable part for intelligent telecommunication networks. Nevertheless, due to the frequent user mobility and complex network scheduling mechanisms, cellular traffic often inherits complicated spatialtemporal patterns, making the prediction incredibly challenging. Although recent advanced algorithms such as graph-based prediction approaches have been proposed, they frequently model spatial dependencies based on static or dynamic graphs and neglect the coexisting multiple spatial correlations induced by traffic generation. Meanwhile, some works lack the consideration of the diverse cellular traffic patterns, result in suboptimal prediction results. In this paper, we propose a novel deep learning network architecture, Adaptive Hybrid Spatial-Temporal Graph Neural Network (AHSTGNN), to tackle the cellular traffic prediction problem. First, we apply adaptive hybrid graph learning to learn the compound spatial correlations among cell towers. Second, we implement a Temporal Convolution Module with multi-periodic temporal data input to capture the nonlinear temporal dependencies. In addition, we introduce an extra Spatial-Temporal Adaptive Module to conquer the heterogeneity lying in cell towers. Our experiments on two real-world cellular traffic datasets show AHSTGNN outperforms the state-of-the-art by a significant margin, illustrating the superior scalability of our method for spatial-temporal cellular traffic prediction.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


In fact, one of the fastest methods for Subgraph Isomorphism works via an arithmetic circuit for evaluating the homomorphism polynomial #REFR , allowing for the randomized detection of a subgraph H with |V (H)| = k and of treewidth w in time O(2 k n w+1 ).

In our first application in this section, we note a generalisation of the above results. The second result regards subgraph isomorphism.

Given graphs G and H, the problem of checking whether H is a subgraph of G parameterized by |V (H)| can be either FPT, as when H is a path, or W[1]-hard, as when H is a clique.

More generally, Subgraph Isomorphism is FPT by |V (H)| if H comes from a family of graphs with bounded treewidth (originally shown using the colour-coding technique of Alon et al.

#OTHEREFR ), and there is good evidence that no more general such class exists #OTHEREFR .

In general, the parameterized complexity of subgraph isomorphism problems has been extensively and meticulously investigated #OTHEREFR .

Furthermore, the exponent w + 1 here is optimal, up to plausible conjectures #OTHEREFR .

We observe that this running time is compatible with an additional constraint that the copy of H found in G should be independent in a given linear matroid.

Determinantal Sieving

Abstract: We introduce a new, remarkably powerful tool to the toolbox of algebraic FPT algorithms, determinantal sieving. Given a polynomial P (x 1 ,. .. , x n) over a field F of characteristic 2, on a set of variables X = {x 1 ,. .. , x n }, and a linear matroid M = (X, I) over F of rank k, in 2 k evaluations of P we can sieve for those terms in the monomial expansion of P which are multilinear and whose support is a basis for M. The known tools of multilinear detection and constrained multilinear detection then correspond to the case where M is a uniform matroid, respectively the truncation of a disjoint union of uniform matroids. More generally, let the odd support of a monomial m be the set of variables which have odd degree in m. Using 2 k evaluations of P , we can sieve for those terms m whose odd support spans M. Applying this framework to well-known efficiently computable polynomial families allows us to simplify, generalize and improve on a range of algebraic FPT algorithms, such as:

Faster Algorithms For Finding And Counting Subgraphs

Abstract: In this paper we study a natural generalization of both {\sc $k$-Path} and {\sc $k$-Tree} problems, namely, the {\sc Subgraph Isomorphism} problem. In the {\sc Subgraph Isomorphism} problem we are given two graphs $F$ and $G$ on $k$ and $n$ vertices respectively as an input, and the question is whether there exists a subgraph of $G$ isomorphic to $F$. We show that if the treewidth of $F$ is at most $t$, then there is a randomized algorithm for the {\sc Subgraph Isomorphism} problem running in time $\cO^*(2^k n^{2t})$. To do so, we associate a new multivariate {Homomorphism polynomial} of degree at most $k$ with the {\sc Subgraph Isomorphism} problem and construct an arithmetic circuit of size at most $n^{\cO(t)}$ for this polynomial. Using this polynomial, we also give a deterministic algorithm to count the number of homomorphisms from $F$ to $G$ that takes $n^{\cO(t)}$ time and uses polynomial space. For the counting version of the {\sc Subgraph Isomorphism} problem, where the objective is to count the number of distinct subgraphs of $G$ that are isomorphic to $F$, we give a deterministic algorithm running in time and space $\cO^*({n \choose k/2}n^{2p})$ or ${n\choose k/2}n^{\cO(t \log k)}$. We also give an algorithm running in time $\cO^{*}(2^{k}{n \choose k/2}n^{5p})$ and taking space polynomial in $n$. Here $p$ and $t$ denote the pathwidth and the treewidth of $F$, respectively. Thus our work not only improves on known results on {\sc Subgraph Isomorphism} but it also extends and generalize most of the known results on {\sc $k$-Path} and {\sc $k$-Tree}.


A DAE is a type of DNN that is trained to detect and discard noise from input data and reconstruct it back to its original form #REFR .

Several defence strategies have been proposed to overcome this problem, including defensive distillation, which has been successful in defending DNNs from adversarial attacks in run-time settings #OTHEREFR .

Nevertheless, one of the drawbacks of the defensive distillation method is that it remains susceptible to data poisoning attacks, in which adversaries aim to impair the model's performance by inserting erroneous data entries into the training set.

These harmful data points could be carefully crafted to be close to the model's decision boundary to circumvent the countermeasures offered by defensive distillation.

This study presents a novel method for robustifying a distilled network against data poisoning adversarial threats by integrating a denoising autoencoder (DAE) #OTHEREFR in the defensive distillation cycle.

Defensive distillation involves training two DNNs, the instructor model (the first model), and distilling its knowledge into the student model (the second or distilled model) to make it robust adversarial examples and previously unseen input #OTHEREFR .

This paper is motivated by the fact that the instructor model is not immune to data poisoning adversarial attacks.

Although the student model has more latitude to reject input modifications because it leverages the "distilled" version of the training data, where the training examples are transformed by a temperature parameter T #OTHEREFR .

Thus, minimising the instructor model's susceptibility to data poisoning attacks is pivotal for developing a reliable, distilled DNN.

To achieve this, we designed a DAE to detect and reconstruct poisonous adversarial inputs in the training data.

The defensive distillation method already offers a strong foundation, but when combined with a DAE, the protection mechanism against data poisoning adversarial attacks is significantly strengthened.

Denoising Autoencoder-Based Defensive Distillation As An Adversarial Robustness Algorithm

Abstract: Adversarial attacks significantly threaten the robustness of deep neural networks (DNNs). Despite the multiple defensive methods employed, they are nevertheless vulnerable to poison attacks, where attackers meddle with the initial training data. In order to defend DNNs against such adversarial attacks, this work proposes a novel method that combines the defensive distillation mechanism with a denoising autoencoder (DAE). This technique tries to lower the sensitivity of the distilled model to poison attacks by spotting and reconstructing poisonous adversarial inputs in the training data. We added carefully created adversarial samples to the initial training data to assess the proposed method's performance. Our experimental findings demonstrate that our method successfully identified and reconstructed the poisonous inputs while also considering enhancing the DNN's resilience. The proposed approach provides a potent and robust defence mechanism for DNNs in various applications where data poisoning attacks are a concern. Thus, the defensive distillation technique's limitation posed by poisonous adversarial attacks is overcome.

Autoencoders

Abstract: An autoencoder is a specific type of a neural network, which is mainlydesigned to encode the input into a compressed and meaningful representation, andthen decode it back such that the reconstructed input is similar as possible to theoriginal one. This chapter surveys the different types of autoencoders that are mainlyused today. It also describes various applications and use-cases of autoencoders.


In #REFR , a function-aggregation method is proposed, which assumes that one utility function can capture only a part of users' preferences, and the whole preferences can be infinitely approximated by introducing enough utility functions.

Recently, many machine learning-based methods have been proposed to specially address choice problems.

These methods do not need an extra transformation process and avoid possible preference distortions.

For example, the pointer neural network (PNN) #OTHEREFR , which is an encoder-decoder network based on recurrent neural network and attention mechanism, is utilized as the mapping function between the available items in the markets and users'

The available items are fed into PNN one by one, and PNN would point to the item that is predicted as the best choice #OTHEREFR .

The function-aggregation method adopts multiple utility functions simultaneously, and the total utility is equal to their weighted sum.

The restricted Boltzmann machine is adopted in #OTHEREFR to model context effects.

It is proved that the restricted Boltzmann machine can be translated into the MNL formulation, together with an additional term accounting for the comparisons among items.

Pacos: Modeling Users' Interpretable And Context-Dependent Choices In Preference Reversals

Abstract: Choice problems refer to selecting the best choices from several items, and learning users' preferences in choice problems is of great significance in understanding the decision making mechanisms and providing personalized services. Existing works typically assume that people evaluate items independently. In practice, however, users' preferences depend on the market in which items are placed, which is known as context effects; and the order of users' preferences for two items may even be reversed, which is referred to preference reversals. In this work, we identify three factors contributing to context effects: users' adaptive weights, the inter-item comparison, and display positions. We propose a context-dependent preference model named Pacos as a unified framework for addressing three factors simultaneously, and consider two design methods including an additive method with high interpretability and an ANN-based method with high accuracy. We study the conditions for preference reversals to occur and provide an theoretical proof of the effectiveness of Pacos in addressing preference reversals. Experimental results show that the proposed method has better performance than prior works in predicting users' choices, and has great interpretability to help understand the cause of preference reversals.

Predicting Choice With Set-Dependent Aggregation

Abstract: Providing users with alternatives to choose from is an essential component in many online platforms, making the accurate prediction of choice vital to their success. A renewed interest in learning choice models has led to significant progress in modeling power, but most current methods are either limited in the types of choice behavior they capture, cannot be applied to large-scale data, or both. ::: Here we propose a learning framework for predicting choice that is accurate, versatile, theoretically grounded, and scales well. Our key modeling point is that to account for how humans choose, predictive models must capture certain set-related invariances. Building on recent results in economics, we derive a class of models that can express any behavioral choice pattern, enjoy favorable sample complexity guarantees, and can be efficiently trained end-to-end. Experiments on three large choice datasets demonstrate the utility of our approach.


We define this threat model according to Carlini et al #REFR by defining the goals, capabilities, and knowledge of the target AI system, that the adversary has.

In order to have correct metrics, and to assess the validity of our approach in a real-world context, it is essential that we define a threat model using a precise taxonomy.

Let C(x) : X → Y be a classifier, and x ∈ X ⊂ R d an input to the classifier.

Let y true ∈ Y = {1, 2, ..., N } be the ground truth of the input x, i.e. the true class label of x among N classes.

We call y pred = C(x) the predicted label for x.

In our study, we perform untargeted attacks, ie.

the adversary's goal is to generate an adversarial example x adv for x that is misclassified by C.

Evaluating Adversarial Robustness On Document Image Classification

Abstract: Adversarial attacks and defenses have gained increasing interest on computer vision systems in recent years, but as of today, most investigations are limited to natural images. However, many artificial intelligence models actually handle documentary data, which is very different from real world images. Hence, in this work, we try to apply the adversarial attack philosophy on documentary data and to protect models against such attacks. Our methodology is to implement untargeted gradient-based, transfer-based and score-based attacks and evaluate the impact of defenses such as adversarial training, JPEG input compression and grey-scale input transformation on the robustness of ResNet50 and EfficientNetB0 model architectures. To the best of our knowledge, no such work has been conducted by the community in order to study the impact of these attacks on the document image classification task.

On Evaluating Adversarial Robustness

Abstract: Correctly evaluating defenses against adversarial examples has proven to be extremely difficult. Despite the significant amount of recent work attempting to design defenses that withstand adaptive attacks, few have succeeded; most papers that propose defenses are quickly shown to be incorrect. ::: We believe a large contributing factor is the difficulty of performing security evaluations. In this paper, we discuss the methodological foundations, review commonly accepted best practices, and suggest new methods for evaluating defenses to adversarial examples. We hope that both researchers developing defenses as well as readers and reviewers who wish to understand the completeness of an evaluation consider our advice in order to avoid common pitfalls.


One of the most representative GAN-based methods is CycleGAN #REFR , which uses cyclic consistency losses to constrain the entire model into two generative networks.

The primary idea behind the image-to-image translation task is to establish a mapping between images from two different domains.

In medical image translation, the main connection between two domains is the underlying anatomical structure information.

The previous works on the image translation are mainly based on GANs #OTHEREFR and VAEs #OTHEREFR .

It can translate a source domain image to the target domain and vice versa.

Another representative work is the Geometry-consistent generative adversarial network (GcGAN) #OTHEREFR , which provides unilateral unsupervised mapping.

GcGAN maps the original image into two different predefined geometric transformations and generates two images in a new domain under the corresponding geometric consistency constraints.

However, GAN-based models have some inherent drawbacks since they are trained by two networks (generator and discriminator) playing against each other, which can render the training difficult and unstable.

For VAE-based methods, one of the most representative works on unpaired image translation is UNIT #OTHEREFR , which assumes that two domains share a common latent space and the corresponding images in both domains are mapped to the same latent code.

Zero-Shot Medical Image Translation Via Frequency-Guided Diffusion Models

Abstract: Recently, the diffusion model has emerged as a superior generative model that can produce highquality images with excellent realism. There is a growing interest in applying diffusion models to image translation tasks. However, for medical image translation, the existing diffusion models are deficient in accurately retaining structural information since the structure details of source domain images are lost during the forward diffusion process and cannot be fully recovered through learned reverse diffusion, while the integrity of anatomical structures is extremely important in medical images. Training and conditioning diffusion models using paired source and target images with matching anatomy can help. However, such paired data are very difficult and costly to obtain, and may also reduce the robustness of the developed model to outof-distribution testing data. We propose a frequency-guided diffusion model (FGDM) that employs frequency-domain filters to guide the diffusion model for structure-preserving image translation. Based on its design, FGDM allows zeroshot learning, as it can be trained solely on the data from the target domain, and used directly for source-to-target domain translation without any exposure to the sourcedomain data during training. We trained FGDM solely on the head-and-neck CT data, and evaluated it on both headand-neck and lung cone-beam CT (CBCT)-to-CT translation tasks. FGDM outperformed the state-of-the-art methods (GAN-based, VAE-based, and diffusion-based) in all metrics, showing its significant advantages in zero-shot medical image translation.

Unpaired Image-To-Image Translation Using Cycle-Consistent Adversarial Networks

Abstract: Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. We present an approach for learning to translate an image from a source domain $X$ to a target domain $Y$ in the absence of paired examples. Our goal is to learn a mapping $G: X \rightarrow Y$ such that the distribution of images from $G(X)$ is indistinguishable from the distribution $Y$ using an adversarial loss. Because this mapping is highly under-constrained, we couple it with an inverse mapping $F: Y \rightarrow X$ and introduce a cycle consistency loss to push $F(G(X)) \approx X$ (and vice versa). Qualitative results are presented on several tasks where paired training data does not exist, including collection style transfer, object transfiguration, season transfer, photo enhancement, etc. Quantitative comparisons against several prior methods demonstrate the superiority of our approach.


The networks use a softmax classifier for person identification, whereas a sigmoid layer for soft-biometrics identification #REFR .

OBDs from each human limb is allotted a branch of four convolutional layers.

The convolutional blocks extract descriptive local features from the input OBD data, while the subsequent layers assimilate the global view of the extracted features.

Prior to concatenation, the outputs of each convolutional blocks are processed by a fully connected layer or LSTM layer depending on the network type.

Concatenation is followed by a two-layered fully connected MLP and a classifier layer for tCNN-IMU M LP .

In the case of tCNN-IMU LST M , the concatenation is followed by two LSTM layers and a classifier layer.

Soft-biometrics of individuals describe or categorise an individual or a group of individuals #OTHEREFR , e.g., Gender Identity, Age, Weight and Height.

Attribute representations is a method of describing the data semantically #OTHEREFR . An attribute vector a represents a set of soft-biometrics.

A similar combination of soft-biometrics could represent different persons with similar features.

The Nearest Neighbour Approach (NNA) is used for soft-biometrics-based identification.

The NNA calculates the distance between a prediction attribute vector a and an attribute representation A, with all the different combinations of soft-biometrics.

Multi-Channel Time-Series Person And Soft-Biometric Identification

Abstract: Multi-channel time-series datasets are popular in the context of human activity recognition (HAR). On-body device (OBD) recordings of human movements are often preferred for HAR applications not only for their reliability but as an approach for identity protection, e.g., in industrial settings. Contradictory, the gait activity is a biometric, as the cyclic movement is distinctive and collectable. In addition, the gait cycle has proven to contain soft-biometric information of human groups, such as age and height. Though general human movements have not been considered a biometric, they might contain identity information. This work investigates person and soft-biometrics identification from OBD recordings of humans performing different activities using deep architectures. Furthermore, we propose the use of attribute representation for soft-biometric identification. We evaluate the method on four datasets of multi-channel time-series HAR, measuring the performance of a person and soft-biometrics identification and its relation concerning performed activities. We find that person identification is not limited to gait activity. The impact of activities on the identification performance was found to be training and dataset specific. Soft-biometric based attribute representation shows promising results and emphasis the necessity of larger datasets.

Learning Attribute Representation For Human Activity Recognition

Abstract: Attribute representations became relevant in image recognition and word spotting, providing support under the presence of unbalance and disjoint datasets. However, for human activity recognition using sequential data from on-body sensors, human-labeled attributes are lacking. This paper introduces a search for attributes that represent favorably signal segments for recognizing human activities. It presents three deep architectures, including temporal-convolutions and an IMU centered design, for predicting attributes. An empiric evaluation of random and learned attribute representations, and as well as the networks is carried out on two datasets, outperforming the state-of-the art.

soft-biometrics identification
softmax classifier


Due to these reasons, techniques such as ML and deep learning created a new research direction in the financial literature #REFR .

Statistical models make some simplifying assumptions to be able to obtain some theoretical guarantees and thus have a simple structure and potentially lover performance compared to machine learning techniques.

Machine learning models are more general, make fewer simplifying assumptions, and offer better model-fitting capabilities.

To forecast the future value of financial assets and find the reason for these assets' behavior, numerous machine learning (ML) techniques were employed, such as SVM #OTHEREFR , Support Vector Regression (SVR) #OTHEREFR , Random Forest (RF) #OTHEREFR , and convolutional neural networks (CNN) #OTHEREFR .

These works show that it is possible to use ML techniques to make more accurate forecasts and as alternative approaches to conventional techniques for investigating the behavior of financial assets. #OTHEREFR

(2019) applied several ML techniques such as single hidden layer feed-forward neural network, MLP, autoencoders, and bagof-features algorithms to forecast the mid-price movement of the stock price.

The results indicated that the mentioned ML techniques could forecast price movement. #OTHEREFR

(2019) organized the stock price prediction techniques into four categories, namely statistical methods, pattern recognition, machine learning, and sentiment analysis.

Feature Selection For Forecasting *

Abstract: This work investigates the importance of feature selection for improving the forecasting performance of machine learning algorithms for financial data. Artificial neural networks (ANN), convolutional neural networks (CNN), long-short term memory (LSTM) networks, as well as linear models were applied for forecasting purposes. The Feature Selection with Annealing (FSA) algorithm was used to select the features from about 1000 possible predictors obtained from 26 technical indicators with specific periods and their lags. In addition to this, the Boruta feature selection algorithm was applied as a baseline feature selection method. The dependent variables consisted of daily logarithmic returns and daily trends of ten financial data sets, including cryptocurrency and different stocks. Experiments indicate that the FSA algorithm increased the performance of ML models regardless of the problem type. The FSA hybrid machine learning models showed better performance in 10 out of 10 data sets for regression and 8 out of 10 data sets for classification. None of the hybrid Boruta models outperformed the hybrid FSA models. However, the BOR-CNN model performance was comparable to the best model for 4 out of 10 data sets for regression estimates. BOR-LR and BOR-CNN models showed comparable performance with the best hybrid FSA models in 2 out of 10 datasets for classification. FSA was observed to improve the model performance in both better performance metrics as well as a decreased computation time by providing a lower dimensional input feature space.

Ascertaining Price Formation In Cryptocurrency Markets With Deeplearning

Abstract: The cryptocurrency market is amongst the fastest-growing of all the financial markets in the world. Unlike traditional markets, such as equities, foreign exchange and commodities, cryptocurrency market is considered to have larger volatility and illiquidity. This paper is inspired by the recent success of using deep learning for stock market prediction. In this work, we analyze and present the characteristics of the cryptocurrency market in a high-frequency setting. In particular, we applied a deep learning approach to predict the direction of the mid-price changes on the upcoming tick. We monitored live tick-level data from $8$ cryptocurrency pairs and applied both statistical and machine learning techniques to provide a live prediction. We reveal that promising results are possible for cryptocurrencies, and in particular, we achieve a consistent $78\%$ accuracy on the prediction of the mid-price movement on live exchange rate of Bitcoins vs US dollars.


To generate 2 22 reduced samples, 2 #REFR /n matrices must be reduced (one n × n matrix produces 2n reduced samples, see § 4.1).

The total cost of PICANTE is the sum of the resources needed to preprocess data, train the model, and recover the secret.

Data preprocessing is the most resource intensive part of PICANTE.

As the dimension increases, the number of matrices scales down linearly.

To avoid the exponential cost of BKZ-reduction #OTHEREFR , we fix the block size at most β = 20 so that the preprocessing step scales polynomially with n and log q.

See discussion of the parameter choices for preprocessing in § 5.4 below.

In practice, to save resources, we choose smaller β for larger dimensions.

Table 5 reports the preprocessing resources (in cpu hours) for each n. Our preprocessing is fully parallelizable.

Salsa Picante: A Machine Learning Attack On Lwe With Binary Secrets

Abstract: The Learning With Errors (LWE) problem is one of the major hard problems in post-quantum cryptography. For example, 1) the only Key Exchange Mechanism KEM standardized by NIST [14] is based on LWE; and 2) current publicly available Homomorphic Encryption (HE) libraries are based on LWE. NIST KEM schemes use random secrets, but homomorphic encryption schemes use binary or ternary secrets, for efficiency reasons. In particular, sparse binary secrets have been proposed, but not standardized [2], for HE. Prior work SALSA [49] demonstrated a new machine learning attack on sparse binary secrets for the LWE problem in small dimensions (up to n = 128) and low Hamming weights (up to h = 4). However, this attack assumed access to millions of LWE samples, and was not scaled to higher Hamming weights or dimensions. Our attack, PICANTE, reduces the number of samples required to just m = 4n samples. Moreover, it can recover secrets with much larger dimensions (up to 350) and Hamming weights (roughly n/10, or h = 33 for n = 300). To achieve this, we introduce a preprocessing step which allows us to generate the training data from a linear number of samples and changes the distribution of the training data to improve transformer training. We also improve the distinguisher/secret recovery methods of SALSA and introduce a novel cross-attention recovery mechanism which allows us to read-off the secret directly from the trained models.

Universal Transformers

Abstract: Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been shown to achieve superior results on some sequence modeling tasks such as machine translation, with the added advantage that they concurrently process all inputs in the sequence, leading to easy parallelization and faster training times. Despite these successes, however, popular feed-forward sequence models like the Transformer fail to generalize in many simple tasks that recurrent models handle with ease, e.g. copying strings or even simple logical inference when the string or formula lengths exceed those observed at training time. We propose the Universal Transformer (UT), a parallel-in-time self-attentive recurrent sequence model which can be cast as a generalization of the Transformer model and which addresses these issues. UTs combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs. We also add a dynamic per-position halting mechanism and find that it improves accuracy on several tasks. In contrast to the standard Transformer, under certain assumptions, UTs can be shown to be Turing-complete. Our experiments show that UTs outperform standard Transformers on a wide range of algorithmic and language understanding tasks, including the challenging LAMBADA language modeling task where UTs achieve a new state of the art, and machine translation where UTs achieve a 0.9 BLEU improvement over Transformers on the WMT14 En-De dataset.


This result aligns with the intuition that the Fourier tempogram is the most similar to the CQT representation in #REFR as it tends to have upper harmonics but not sub-harmonics.

Since calibration results correlate with model performance, i.e.

a model with more linear and less disperse calibration curves leads to more consistent estimations #OTHEREFR , we look at calibration results for each model, depicted in Figure 3 .

The Fourier tempogram shows more consistent calibration results across data distributions and leads to smoother curves for both synthetic and real data.

On the contrary, the autocorrelation tempogram results present the most variation, meaning that for similar tempo values, the model struggles to assign the same tempo output.

Surprisingly, the hybrid tempogram does worse than the Fourier tempogram, which suggests that the multiplication of the autocorrelation and Fourier tempograms removes harmonic information that the model relies on to perform tempo estimation.

The experiments with the different data distributions show that the model does not benefit from the wider range of a log-uniform distribution, and instead is able to extrapolate from a log-normal distribution centered in the lower tempi end (70 bpm).

When we look at this result closely, an explanation may be found in Figure 2 .

Given a tempo distribution expressed in linear BPM values (Figure 2 top), the model will be fed a slightly different tempo distribution ( Figure  2 bottom) because of the logarithmic effect of the input tempogram.

Tempo Vs. Pitch: Understanding Self-Supervised Tempo Estimation

Abstract: Self-supervision methods learn representations by solving pretext tasks that do not require human-generated labels, alleviating the need for time-consuming annotations. These methods have been applied in computer vision, natural language processing, environmental sound analysis, and recently in music information retrieval, e.g. for pitch estimation. Particularly in the context of music, there are few insights about the fragility of these models regarding different distributions of data, and how they could be mitigated. In this paper, we explore these questions by dissecting a self-supervised model for pitch estimation adapted for tempo estimation via rigorous experimentation with synthetic data. Specifically, we study the relationship between the input representation and data distribution for self-supervised tempo estimation.

Spice: Self-Supervised Pitch Estimation

Abstract: We propose a model to estimate the fundamental frequency in monophonic audio, often referred to as pitch estimation. We acknowledge the fact that obtaining ground truth annotations at the required temporal and frequency resolution is a particularly daunting task. Therefore, we propose to adopt a self-supervised learning technique, which is able to estimate (relative) pitch without any form of supervision. The key observation is that pitch shift maps to a simple translation when the audio signal is analysed through the lens of the constant-Q transform (CQT). We design a self-supervised task by feeding two shifted slices of the CQT to the same convolutional encoder, and require that the difference in the outputs is proportional to the corresponding difference in pitch. In addition, we introduce a small model head on top of the encoder, which is able to determine the confidence of the pitch estimate, so as to distinguish between voiced and unvoiced audio. Our results show that the proposed method is able to estimate pitch at a level of accuracy comparable to fully supervised models, both on clean and noisy audio samples, yet it does not require access to large labeled datasets


To ensure that this oracle has query time O(log h), we generalize the argument of Wulff-Nilsen #REFR from full oracles to partial ones.

The two paths P u,u , P v,v have the property that

Intuitively, given a query (u, v), the partial PRDO provides a u − v path P u,v (rather than paths P u,u , P v,v ) if and only if the Thorup-Zwick oracle does so while employing only the first h levels of the oracle.

We note that using the first h levels of the TZ oracle indeed guarantees almost all the aforementioned properties.

The only problem is that the query time of such partial oracle is O(h), rather than O(log h).

In the second step, to find a low-stretch path between u and v , we construct an interactive emulator for the set S.

An interactive emulator is an oracle that similarly to PRDOs, provides approximate shortest paths between two queried vertices.

However, as opposed to a PRDO, the provided path does not use actual edges in the original graph, but rather employs virtual edges that belong to some sparse low-stretch emulator of the graph.

Our construction of interactive emulator is based on the distance oracle of Mendel and Naor #OTHEREFR . Recall that the Mendel-Naor oracle is not path-reporting.

Built for N -vertex graph and a parameter k 1 , this oracle provides stretch O(k 1 ), has size O(N 1+ 1 k 1 ), and has query time O(1).

Path-Reporting Distance Oracles With Near-Logarithmic Stretch And Linear Size

Abstract: Given an n-vertex undirected graph G = (V, E, w), and a parameter k ≥ 1, a pathreporting distance oracle (or PRDO) is a data structure of size S(n, k), that given a query (u, v) ∈ V 2 , returns an f (k)-approximate shortest u − v path P in G within time q(k) + O(|P |). Here S(n, k), f (k) and q(k) are arbitrary (hopefully slowly-growing) functions. A distance oracle that only returns an approximate estimated(u, v) of the distance d G (u, v) between the queried vertices is called a non-path-reporting distance oracle. A landmark PRDO due to Thorup and Zwick [54] has S(n, k) = O(k • n 1+ 1 k), f (k) = 2k − 1 and q(k) = O(k). Wulff-Nilsen [57] devised an improved query algorithm for this oracle with q(k) = O(log k). The size of this oracle is Ω(n log n) for all k. Elkin and Pettie [28] devised a PRDO with S(n, k) = O(log k • n 1+ 1 k), f (k) = O(k log 4/3 7) and q(k) = O(log k). Neiman and Shabat [44] recently devised an improved PRDO with S(n, k) = O(n 1+ 1 k), f (k) = O(k log 4/3 4) and q(k) = O(log k). These oracles (of [28, 44]) can be much sparser than O(n log n) (the oracle of [44] can have linear size), but their stretch is polynomially larger than the optimal bound of 2k − 1. On the other hand, a long line of non-path-reporting distance oracles culminated in a celebrated result by Chechik [12], in which S(n, k) = O(n 1+ 1 k), f (k) = 2k − 1 and q(k) = O(1). In this paper we make a dramatic progress in bridging the gap between path-reporting and non-path-reporting distance oracles. In particular, we devise a PRDO with size S(n, k) = O k log log n log n

Approximate Distance Oracles With Improved Query Time

Abstract: Given an undirected graph $G$ with $m$ edges, $n$ vertices, and non-negative edge weights, and given an integer $k\geq 2$, we show that a $(2k-1)$-approximate distance oracle for $G$ of size $O(kn^{1 + 1/k})$ and with $O(\log k)$ query time can be constructed in $O(\min\{kmn^{1/k},\sqrt km + kn^{1 + c/\sqrt k}\})$ time for some constant $c$. This improves the $O(k)$ query time of Thorup and Zwick. Furthermore, for any $0<\epsilon \leq 1$, we give an oracle of size $O(kn^{1 + 1/k})$ that answers $((2 + \epsilon)k)$-approximate distance queries in $O(1/\epsilon)$ time. At the cost of a $k$-factor in size, this improves the $128k$ approximation achieved by the constant query time oracle of Mendel and Naor and approaches the best possible tradeoff between size and stretch, implied by a widely believed girth conjecture of Erd\H{o}s. We can match the $O(n^{1 + 1/k})$ size bound of Mendel and Naor for any constant $\epsilon>0$ and $k = O(\log n/\log\log n)$.


This is somewhat contrary to the results obtained in LDP paper #REFR , but it is apparently an advantage of RF, since it considers each feature separately, while calculating tree splits.

Additionally, for each value, we also calculate the absolute average difference between its result and the best result for a given hyperparameter on each dataset.

The lower the absolute difference, the better, since it means that a given hyperparameter value, on average, gives the best results among all its possible values. results are presented in Table 3 .

For the number of bins, we can clearly select 50 bins as the optimal value.

While 30 bins gave the best results the same number of times, on average they performed worse compared to the optimal hyperparameter value.

Similarly, for normalization it is evident that we do not need to perform any kind of normalization, since using no normalization obtained the best results on majority of datasets and on average.

For aggregation method, the results are very close, both for number of wins and average difference compared to the best result.

In this case, the choice does not matter that much, and we choose the simpler histogram method.

The linear scale obtained much better results on average than the log scale, so the choice is obvious.

Overall, this means that we can confidently recommend default values for all LDP hyperparameters, and tuning them is not particularly helpful.

This dramatically decreases the computational cost, while having little effect on accuracy on average, which is a desirable tradeoff in a baseline method. Table 3 .

Strengthening Structural Baselines For Graph Classification Using Local Topological Profile

Abstract: We present the analysis of the topological graph descriptor Local Degree Profile (LDP), which forms a widely used structural baseline for graph classification. Our study focuses on model evaluation in the context of the recently developed fair evaluation framework, which defines rigorous routines for model selection and evaluation for graph classification, ensuring reproducibility and comparability of the results. Based on the obtained insights, we propose a new baseline algorithm called Local Topological Profile (LTP), which extends LDP by using additional centrality measures and local vertex descriptors. The new approach provides the results outperforming or very close to the latest GNNs for all datasets used. Specifically, state-of-the-art results were obtained for 4 out of 9 benchmark datasets. We also consider computational aspects of LDP-based feature extraction and model construction to propose practical improvements affecting execution speed and scalability. This allows for handling modern, large datasets and extends the portfolio of benchmarks used in graph representation learning. As the outcome of our work, we obtained LTP as a simple to understand, fast and scalable, still robust baseline, capable of outcompeting modern graph classification models such as Graph Isomorphism Network (GIN). We provide open-source implementation at GitHub.

A Simple Yet Effective Baseline For Non-Attribute Graph Classification

Abstract: Graphs are complex objects that do not lend themselves easily to typical learning tasks. Recently, a range of approaches based on graph kernels or graph neural networks have been developed for graph classification and for representation learning on graphs in general. As the developed methodologies become more sophisticated, it is important to understand which components of the increasingly complex methods are necessary or most effective. As a first step, we develop a simple yet meaningful graph representation, and explore its effectiveness in graph classification. We test our baseline representation for the graph classification task on a range of graph datasets. Interestingly, this simple representation achieves similar performance as the state-of-the-art graph kernels and graph neural networks for non-attributed graph classification. Its performance on classifying attributed graphs is slightly weaker as it does not incorporate attributes. However, given its simplicity and efficiency, we believe that it still serves as an effective baseline for attributed graph classification. Our graph representation is efficient (linear-time) to compute. We also provide a simple connection with the graph neural networks. Note that these observations are only for the task of graph classification while existing methods are often designed for a broader scope including node embedding and link prediction. The results are also likely biased due to the limited amount of benchmark datasets available. Nevertheless, the good performance of our simple baseline calls for the development of new, more comprehensive benchmark datasets so as to better evaluate and analyze different graph learning methods. Furthermore, given the computational efficiency of our graph summary, we believe that it is a good candidate as a baseline method for future graph classification (or even other graph learning) studies.


The dataset we used was the ReClor #REFR training and validation sets, which contain 5138 samples.

Each data point has a context, a question, and 4 answer choices.

Exactly 1 answer choice is correct; each of the other choices is either logically or factually inaccurate.

The ReClor questions are sourced from past GMAT and LSAT exams and high-quality practice exams.

Polytuplet Loss: A Reverse Approach To Training Reading Comprehension And Logical Reasoning Models

Abstract: Throughout schooling, students are tested on reading comprehension and logical reasoning. Students have developed various strategies for completing such exams, some of which are generally thought to outperform others. One such strategy involves emphasizing relative accuracy over absolute accuracy and can theoretically produce the correct answer without full knowledge of the information required to solve the question. This paper examines the effectiveness of applying such a strategy to train transfer learning models to solve reading comprehension and logical reasoning questions. The models were evaluated on the ReClor dataset, a challenging reading comprehension and logical reasoning benchmark. While previous studies targeted logical reasoning skills, we focus on a general training method and model architecture. We propose the polytuplet loss function, an extension of the triplet loss function, to ensure prioritization of learning the relative correctness of answer choices over learning the true accuracy of each choice. Our results indicate that models employing polytuplet loss outperform existing baseline models. Although polytuplet loss is a promising alternative to other contrastive loss functions, further research is required to quantify the benefits it may present. 1

Reclor: A Reading Comprehension Dataset Requiring Logical Reasoning

Abstract: Recent powerful pre-trained language models have achieved remarkable performance on most of the popular datasets for reading comprehension. It is time to introduce more challenging datasets to push the development of this field towards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, human-annotated datasets usually contain biases, which are often exploited by models to achieve high accuracy without truly understanding the text. In order to comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest as HARD set. Empirical results show that the state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.


To reduce the computation cost, Monte-Carlo and gradient-based techniques are proposed to approximate data SV when using large dataset to train the complex models #REFR . Jia et al.

And most of efforts have been devoted to the last two directions.

For strategies of data valuation, the commonly-used approach is based on marginal contribution (MC).

As the basic method for data valuation, LOO is used to evaluate the value of training data point X i by observational change of model performance when leaving out that data point from the training dataset.

To overcome inaccuracy and strict desirability of LOO, SV originated from Cooperative Game Theory (CGT) is widely used to measure the contribution of data #OTHEREFR .

Considering the joining sequence of each training data point, SV needs to calculate the marginal performance of all possible subsets in which the time complexity is exponential.

#OTHEREFR propose an efficient approximation approach to estimate the exact SV for K-nearest-neighor (KNN).

Other works also apply these to other fields such as collaboration machine learning #OTHEREFR .

Though several works devotes to accelerate the computation of traditional SV, it is still impossible to use these approaches in practical due to their inevitable expensive computation.

When it comes to performance metric of data valuation, some recent studies start to consider data-driven metric rather than model-driven metric. Tay et al.

#OTHEREFR propose Cosine Gradient Shapley Value (CGSV) which leverages cosine similarity between gradient vectors of the model to evaluate the contribution of each participant in collaborative machine learning.

Matching-Based Data Valuation For Generative Model

Abstract: Data valuation is critical in machine learning, as it helps enhance model transparency and protect data properties. Existing data valuation methods have primarily focused on discriminative models, neglecting deep generative models that have recently gained considerable attention. Similar to discriminative models, there is an urgent need to assess data contributions in deep generative models as well. However, previous data valuation approaches mainly relied on discriminative model performance metrics and required model retraining. Consequently, they cannot be applied directly and efficiently to recent deep generative models, such as generative adversarial networks and diffusion models, in practice. To bridge this gap, we formulate the data valuation problem in generative models from a similarity-matching perspective. Specifically, we introduce Generative Model Valuator (GMValuator), the first model-agnostic approach for any generative models, designed to provide data valuation for generation tasks. We have conducted extensive experiments to demonstrate the effectiveness of the proposed method. To the best of their knowledge, GMValuator is the first work that offers a training-free, post-hoc data valuation strategy for deep generative models. Keywords Generative Model • Data Valuation 1 Introduction Recently, applications of generative models have exploded across many fields. For example, ChatGPT [1] is capable of automatically generating code by utilizing a generative model, and Stable Diffusion [2] can generate photo-realistic images. As a form of unsupervised machine learning, generative models learn the probabilities distribution of the training datasets from which they sample the synthetic data, which is similar to the original training dataset. Data is regarded as the fuel for the model artificial intelligence (i.e., deep learning) [3]. Obtaining high-quality generative models requires a large amount of data, and including informative data samples related to the task is also an important ingredient in the training process of deep generative models. From a data privacy perspective, personal data is protected by various regulations including General Data Protection Regulation (GDPR) [4] and become a valuable asset. Privacy Calculus [5] requires assessing the trade-off between potential benefits and expected costs or risks of disclosing private data. Recent AI Generated Content further arouse people's attention[6]. Therefore, measuring the value of the collected data in generative model training is vital to achieving the training objectives. Unfortunately, there is no existing studies devoting to data valuation for generative model. Most of the previous data valuation methods are for discriminate model [7]. For example, some performance metrics (e.g., loss value, accuracy) of the machine learning model are always used to calculate the score function while the strategy of data valuation is Leave-one-out (LOO), Shapley Value (SV) and Banzhaf Index (BI), etc [8, 9, 10].

Data Shapley: Equitable Valuation Of Data For Machine Learning

Abstract: As data becomes the fuel driving technological and economic growth, a fundamental challenge is how to quantify the value of data in algorithmic predictions and decisions. For example, in healthcare and consumer markets, it has been suggested that individuals should be compensated for the data that they generate, but it is not clear what is an equitable valuation for individual data. In this work, we develop a principled framework to address data valuation in the context of supervised machine learning. Given a learning algorithm trained on $n$ data points to produce a predictor, we propose data Shapley as a metric to quantify the value of each training datum to the predictor performance. Data Shapley value uniquely satisfies several natural properties of equitable data valuation. We develop Monte Carlo and gradient-based methods to efficiently estimate data Shapley values in practical settings where complex learning algorithms, including neural networks, are trained on large datasets. In addition to being equitable, extensive experiments across biomedical, image and synthetic data demonstrate that data Shapley has several other benefits: 1) it is more powerful than the popular leave-one-out or leverage score in providing insight on what data is more valuable for a given learning task; 2) low Shapley value data effectively capture outliers and corruptions; 3) high Shapley value data inform what type of new data to acquire to improve the predictor.


To capture fine-grained interplay between modalities, SCAN #REFR introduce stacked cross attention to enable attention with context from both image and sentence.

The extra label information is employed to boost the discrimination of common representations.

However, the lack of annotations limited the practicability in the real world.

Our proposed approach falls in the category of unsupervised methods.

For example, Wang #OTHEREFR uses a twobranch neural networks to learn the joint embeddings of multimodal data.

Inspired by hard negative mining, VSE++ #OTHEREFR uses hard negatives to improve the retrieval performance.

Recently, motivated by the powerful learning ability of graph model, GSMN #OTHEREFR and SGRAF #OTHEREFR construct graph structure for multimodal data to benefit the learning of finegrained correspondence.

Although existing works have achieved remarkable results, they usually depend on the correct correspondence among multimodal data and cannot tackle the noise issue.

Thus, it is significant to explore how to learn cross-modal retrieval with noisy correspondence, but which is rarely touched in previous studies.

Noisy Correspondence Learning With Meta Similarity Correction

Abstract: Despite the success of multimodal learning in crossmodal retrieval task, the remarkable progress relies on the correct correspondence among multimedia data. However, collecting such ideal data is expensive and timeconsuming. In practice, most widely used datasets are harvested from the Internet and inevitably contain mismatched pairs. Training on such noisy correspondence datasets causes performance degradation because the cross-modal retrieval methods can wrongly enforce the mismatched data to be similar. To tackle this problem, we propose a Meta Similarity Correction Network (MSCN) to provide reliable similarity scores. We view a binary classification task as the meta-process that encourages the MSCN to learn discrimination from positive and negative meta-data. To further alleviate the influence of noise, we design an effective data purification strategy using meta-data as prior knowledge to remove the noisy samples. Extensive experiments are conducted to demonstrate the strengths of our method in both synthetic and real-world noises, including Flickr30K, MS-COCO, and Conceptual Captions. Our code is publicly available. 1

Stacked Cross Attention For Image-Text Matching

Abstract: In this paper, we study the problem of image-text matching. Inferring the latent semantic alignment between objects or other salient stuff (e.g. snow, sky, lawn) and the corresponding words in sentences allows to capture fine-grained interplay between vision and language, and makes image-text matching more interpretable. Prior work either simply aggregates the similarity of all possible pairs of regions and words without attending differentially to more and less important words or regions, or uses a multi-step attentional process to capture limited number of semantic alignments which is less interpretable. In this paper, we present Stacked Cross Attention to discover the full latent alignments using both image regions and words in a sentence as context and infer image-text similarity. Our approach achieves the state-of-the-art results on the MS-COCO and Flickr30K datasets. On Flickr30K, our approach outperforms the current best methods by 22.1% relatively in text retrieval from image query, and 18.2% relatively in image retrieval with text query (based on Recall@1). On MS-COCO, our approach improves sentence retrieval by 17.8% relatively and image retrieval by 16.6% relatively (based on Recall@1 using the 5K test set). Code has been made available at: https://github.com/kuanghuei/SCAN.


In #REFR , the Riemannian Brownian motion model was constructed over the manifold by using VAE (R-VAE).

A normal prior, on the other hand, causes the approximate posterior to be over-regularized, leading to a less effective learned latent representation of the input.

The goal of manifold learning approaches is to infer latent representations that reflect the inherent geometric structure of data.

Such methods are mainly used for data visualization and dimensionality reduction and recently are used to improve the performance of generative models #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR .

Traditional non-linear manifold learning methods like local tangent space alignment #OTHEREFR and linear embedding #OTHEREFR maintain local structures and extract the features of data manifolds.

The Riemannian structure overcomes the identifiability problem by providing a meaningful representation which is not affected by reparametrizations.

However, Riemannian manifold-based models cannot accurately find the shortest path between two samples that are far away from each other.

Generally, Riemannian geometry methods use Ordinary Differential Equation (ODE) to find the shortest interpolation curve in the latent space.

ODE is computationally expensive and cannot find the shortest interpolation curve more consistently than straight-length methods.

The interpolation among data samples is widely used to attain a smooth transformation from one sample to another.

Vtae: Variational Transformer Autoencoder With Manifolds Learning

Abstract: Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables and these models use a non-linear function (generator) to map latent samples into the data space. On the other hand, the non-linearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning. This weak projection, however, can be addressed by a Riemannian metric, and we show that geodesics computation and accurate interpolations between data samples on the Riemannian manifold can substantially improve the performance of deep generative models. In this paper, a Variational spatial-Transformer AutoEncoder (VTAE) is proposed to minimize geodesics on a Riemannian manifold and improve representation learning. In particular, we carefully design the variational autoencoder with an encoded spatial-Transformer to explicitly expand the latent variable model to data on a Riemannian manifold, and obtain global context modelling. Moreover, to have smooth and plausible interpolations while traversing between two different objects' latent representations, we propose a geodesic interpolation network different from the existing models that use linear interpolation with inferior performance. Experiments on benchmarks show that our proposed model can improve predictive accuracy and versatility over a range of computer vision tasks, including image interpolations, and reconstructions.

Variational Autoencoders With Riemannian Brownian Motion Priors

Abstract: Variational Autoencoders (VAEs) represent the given data in a low-dimensional latent space, which is generally assumed to be Euclidean. This assumption naturally leads to the common choice of a standard Gaussian prior over continuous latent variables. Recent work has, however, shown that this prior has a detrimental effect on model capacity, leading to subpar performance. We propose that the Euclidean assumption lies at the heart of this failure mode. To counter this, we assume a Riemannian structure over the latent space, which constitutes a more principled geometric view of the latent codes, and replace the standard Gaussian prior with a Riemannian Brownian motion prior. We propose an efficient inference scheme that does not rely on the unknown normalizing factor of this prior. Finally, we demonstrate that this prior significantly increases model capacity using only one additional scalar parameter.


BYOL method is highly sensitive to the exponential moving average (EMA) value of a momentum encoder #REFR .

We used an Adam optimizer #OTHEREFR with a learning rate of 1e-3 and 2 regularization parameter 1e-6.

Therefore, we followed the same EMA scheduler as in the original implementation #OTHEREFR , of which the momentum starts from 0.996 to 1.

We should also note that we did not use the EMA scheduler for the experiments with short epochs.

SwAV method: We mostly followed the same settings as the SimCLR method, such as an SGD optimizer with a learning rate of 0.1 and weight decay of 1e-4.

In the case of pretraining on ImageNet, we used the temperature value of 0.1 and an epsilon value of 0.02, while we froze the 3,000-dimensional prototypes for one epoch.

For every other cases, we froze the 100-dimensional prototypes for ten epochs.

Coreset Sampling From Open-Set For Fine-Grained Self-Supervised Learning

Abstract: Deep learning in general domains has constantly been extended to domain-specific tasks requiring the recognition of fine-grained characteristics. However, real-world applications for fine-grained tasks suffer from two challenges: a high reliance on expert knowledge for annotation and necessity of a versatile model for various downstream tasks in a specific domain (e.g., prediction of categories, bounding boxes, or pixel-wise annotations). Fortunately, the recent self-supervised learning (SSL) is a promising approach to pretrain a model without annotations, serving as an effective initialization for any downstream tasks. Since SSL does not rely on the presence of annotation, in general, it utilizes the large-scale unlabeled dataset, referred to as an open-set. In this sense, we introduce a novel Open-Set Self-Supervised Learning problem under the assumption that a large-scale unlabeled open-set is available, as well as the fine-grained target dataset, during a pretraining phase. In our problem setup, it is crucial to consider the distribution mismatch between the open-set and target dataset. Hence, we propose SimCore algorithm to sample a coreset, the subset of an open-set that has a minimum distance to the target dataset in the latent space. We demonstrate that SimCore significantly improves representation learning performance through extensive experimental settings, including eleven fine-grained datasets and seven open-sets in various downstream tasks.

Momentum Contrast For Unsupervised Visual Representation Learning

Abstract: We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.


This is in contrast to the results reported in FiFTy #REFR , where the inference time of FiFTy can be significantly larger.

In summary, the results show that all three models have similar inference times, with minimal variance between runs.

Our models are optimized for efficient inference by utilizing cutting-edge techniques to minimize computational overhead and reduce latency.

As a result, we were able to achieve inference times that are comparable to state-of-the-art models while still maintaining high accuracy.

This demonstrates the effectiveness of our proposed models in real-world applications where fast and efficient inference is crucial.

Similar to previous work #OTHEREFR , we observe that files with high entropy are difficult to classify because there is no statistical trace that the convolutional kernel can extract.

Moreover, many files are container types that contain other files as embedded objects, e.g., pdf files that can contain embedded jpg images. As a result, classifiers behave erratically.

File Fragment Classification Using Light-Weight Convolutional Neural Networks

Abstract: In digital forensics, file fragment classification is an important step toward completing file carving process. There exist several techniques to identify the type of file fragments without relying on meta-data, such as using features like header/footer and N-gram to identify the fragment type. Recently, convolutional neural network (CNN) models have been used to build classification models to achieve this task. However, the number of parameters in CNNs tends to grow exponentially as the number of layers increases. This results in a dramatic increase in training and inference time. In this paper, we propose lightweight file fragment classification models based on depthwise separable CNNs. The evaluation results show that our proposed models provide faster inference time with comparable accuracy as compared to the state-of-art CNN based models. In particular, our models were able to achieve an accuracy of 79% on the FFT-75 dataset with nearly 100K parameters and 164M FLOPs, which is 4x smaller and 6x faster than the state-of-the-art classifier in the literature.

Fifty: Large-Scale File Fragment Type Identification Using Neural Networks

Abstract: We present FiFTy, a modern file type identification tool for memory forensics and data carving. In contrast to previous approaches based on hand-crafted features, we design a compact neural network architecture, which uses a trainable embedding space, akin to successful natural language processing models. Our approach dispenses with explicit feature extraction which is a bottleneck in legacy systems. We evaluate the proposed method on a novel dataset with 75 file types - the most diverse and balanced dataset reported to date. FiFTy consistently outperforms all baselines in terms of speed, accuracy and individual misclassification rates. We achieved an average accuracy of 77.5% with processing speed of approx 38 sec/GB, which is better and more than an order of magnitude faster than the previous state-of-the-art tool - Sceadan (69% at 9 min/GB). Our tool and the corresponding dataset are available publicly online.


For the feature extractor, HG-FEN with ResNet-50 as the backbone and Imagenet-pretrained model #REFR as the initialized weights.

All the experiments are implemented using PyTorch and run on a desktop with 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz and a single NVIDIA GeForce RTX 3090 GPU.

we directly apply the publicly available detector of YOLOX #OTHEREFR trained by #OTHEREFR for MOT17, MOT20, and ablation study on MOT17.

We train our HG-FEN model on MOT17 and MOT20, its parameters are updated using the Adam optimizer #OTHEREFR with weight decay of 5 × 10 −4 .

During the training procedure, the initial learning rate is 3.5×10 −4 , input batch size is set as 64 and the resolution of every image is 256 × 128. Total training 120 epochs.

For the tracker, we set four tracklet states, including tentative, confirmed, deleted and lost state.

To initialize a new tracklet, in the first frame of each video sequence we set the tracklet with a confidence score larger than the new tracklet threshold of 0.7 to the confirm state, while subsequent frames in the case where the condition is met will be set to the tentative.

For the lost tracklets, we keep them for 30 frames in case it appears again. We set λ = 0.95 in Eq. #OTHEREFR .

Rt-Track: Robust Tricks For Multi-Pedestrian Tracking

Abstract: Object tracking is divided into single-object tracking (SOT) and multi-object tracking (MOT). MOT aims to maintain the identities of multiple objects across a series of continuous video sequences. In recent years, MOT has made rapid progress. However, modeling the motion and appearance models of objects in complex scenes still faces various challenging issues. In this paper, we design a novel direction consistency method for smooth trajectory prediction (STP-DC) to increase the modeling of motion information and overcome the lack of robustness in previous methods in complex scenes. Existing methods use pedestrian reidentification (Re-ID) to model appearance, however, they extract more background information which lacks discriminability in occlusion and crowded scenes. We propose a hyper-grain feature embedding network (HG-FEN) to enhance the modeling of appearance models, thus generating robust appearance descriptors. We also proposed other robustness techniques, including CF-ECM for storing robust appearance information and SK-AS for improving association accuracy. To achieve state-of-the-art performance in MOT, we propose a robust tracker named Rttrack, incorporating various tricks and techniques. It achieves 79.5 MOTA, 76.0 IDF1 and 62.1 HOTA on the test set of MOT17.Rt-track also achieves 77.9 MOTA, 78.4 IDF1 and 63.3 HOTA on MOT20, surpassing all published methods.

Imagenet Large Scale Visual Recognition Challenge

Abstract: The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the five years of the challenge, and propose future directions and improvements.


We then fuse them by performing convolutions, and utilize the same number of layers of 3D sparse deconvolutions #REFR as those convolutions used in the instance detection module to resume the point-level features hierarchically.

, where C represents the number of instance categories, U represents the number of instances,x andỹ represent coordinate quantization offsets, z represents the height, l, m and q represent 3D bounding box size and ω represents orientation.

Each score in heatmapÊ cij represents the confidence at location (i, j) belonging to the instance class c.

For more details about CenterHead, please refer to #OTHEREFR .

3) Upsample Fusion Module: In upsample fusion module, we aim to resume point-level features by fusing the spatio-temporal features and the instance features.

We directly concatenate the spatio-temporal features and the instance features as the input.

To strengthen the instance information in the final features and maintain more details learned in different layers, we propagate the instance features from multi-resolution point clouds and form the instance pyramid.

After concatenating with the corresponding spatio-temporal features, we inject them into the upsample fusion module.

Finally, the upsample fusion module outputs the point-

, which are then used to determine the moving label of LiDAR points after passing a softmax function.

Insmos: Instance-Aware Moving Object Segmentation In Lidar Data

Abstract: Identifying moving objects is a crucial capability for autonomous navigation, consistent map generation, and future trajectory prediction of objects. In this paper, we propose a novel network that addresses the challenge of segmenting moving objects in 3D LiDAR scans. Our approach not only predicts point-wise moving labels but also detects instance information of main traffic participants. Such a design helps determine which instances are actually moving and which ones are temporarily static in the current scene. Our method exploits a sequence of point clouds as input and quantifies them into 4D voxels. We use 4D sparse convolutions to extract motion features from the 4D voxels and inject them into the current scan. Then, we extract spatio-temporal features from the current scan for instance detection and feature fusion. Finally, we design an upsample fusion module to output point-wise labels by fusing the spatio-temporal features and predicted instance information. We evaluated our approach on the LiDAR-MOS benchmark based on SemanticKITTI and achieved better moving object segmentation performance compared to state-of-theart methods, demonstrating the effectiveness of our approach in integrating instance information for moving object segmentation. Furthermore, our method shows superior performance on the Apollo dataset with a pre-trained model on SemanticKITTI, indicating that our method generalizes well in different scenes. The code and pre-trained models of our method will be released at https://github.com/nubot-nudt/InsMOS.

3D Semantic Segmentation With Submanifold Sparse Convolutional Networks

Abstract: Convolutional networks are the de-facto standard for analyzing spatio-temporal data such as images, videos, and 3D shapes. Whilst some of this data is naturally dense (e.g., photos), many other data sources are inherently sparse. Examples include 3D point clouds that were obtained using a LiDAR scanner or RGB-D camera. Standard "dense" implementations of convolutional networks are very inefficient when applied on such sparse data. We introduce new sparse convolutional operations that are designed to process spatially-sparse data more efficiently, and use them to develop spatially-sparse convolutional networks. We demonstrate the strong performance of the resulting models, called submanifold sparse convolutional networks (SSCNs), on two tasks involving semantic segmentation of 3D point clouds. In particular, our models outperform all prior state-of-the-art on the test set of a recent semantic segmentation competition.

point-level features
3D sparse deconvolutions


The reasons behind a larger learning rate resulting in a smaller generalization error have been explored in #REFR ; Barrett & Dherin (2020).

Further discussions on why a small learning rate leads to a larger generalization error can be found in #OTHEREFR ; Barrett & Dherin (2020).

The complexity of learning trajectory correlates with the generalization error under mild learning rate In Figure 3 , we carry out experiments under various settings.

Each data point in the figure represents the average of three repeated experiments.

The results demonstrate that both the generalization error and C(J t ) increase as the level of label noise is raised (Figure 3 Left).

The another experiments measure C(J t ) and generalization error for different learning rate and discover that C(J t ) can capture the trend generalization error.

Additionally, Appendix A.7 discusses why a larger learning rate can lead to a smaller C(J t ).

Learning Trajectories Are Generalization Indicators

Abstract: This paper aims to investigate the relation between learning trajectories of the Deep Neural Networks (DNNs) and their corresponding generalization capabilities when being optimized with gradient descent and stochastic gradient descent algorithms. In this paper, we construct Linear Approximation Function to model the trajectory information and we propose a new generalization bound with richer trajectory information based on this modelling. Our proposed generalization bound relies on the complexity of learning trajectory and the ratio between the bias and diversity of training set. Experimental results indicate that the proposed method effectively captures the generalization trend across various iterations, learning rates, and label noise levels across entire training process.

Towards Explaining The Regularization Effect Of Initial Large Learning Rate In Training Neural Networks

Abstract: Stochastic gradient descent with a large initial learning rate is widely used for training modern neural net architectures. Although a small initial learning rate allows for faster training and better test performance initially, the large learning rate achieves better generalization soon after the learning rate is annealed. Towards explaining this phenomenon, we devise a setting in which we can prove that a two layer network trained with large initial learning rate and annealing provably generalizes better than the same network trained with a small learning rate from the start. The key insight in our analysis is that the order of learning different types of patterns is crucial: because the small learning rate model first memorizes low-noise, hard-to-fit patterns, it generalizes worse on hard-to-generalize, easier-to-fit patterns than its large learning rate counterpart. This concept translates to a larger-scale setting: we demonstrate that one can add a small patch to CIFAR-10 images that is immediately memorizable by a model with small initial learning rate, but ignored by the model with large learning rate until after annealing. Our experiments show that this causes the small learning rate model's accuracy on unmodified images to suffer, as it relies too much on the patch early on.

smaller generalization error
larger learning rate


However, this is no longer true when these vectors are not orthogonal-in fact, subtracting a best rank-1 approximation from X can even yield a tensor of higher rank #REFR .

One example is in latent variable model learning #OTHEREFR , where the vectors x i appearing in the decomposition are directly related to the model's parameters (for instance, each x i is the mean of a Gaussian component in a mixture model).

If the vectors x i in (1) were orthogonal, then one could retrieve them by resorting to a greedy deflation procedure, first J. H. de M.

Goulart's work was supported by the ANR LabEx CIMI (ANR-11-LABX-0040) within the French Programme "Investissements d'Avenir.".

introduced by Hotelling for matrices #OTHEREFR : a best rank-1 approximation of X is computed and then subtracted from X, and the process is repeated r times.

Algorithmically, each such approximation can be computed by power iteration #OTHEREFR .

In some applications, this can in principle be circumvented by transforming X in such a way that it becomes a rank-r symmetric orthogonal decomposition, as long as r ≤ n.

For instance, in latent variable model learning the eigendecomposition of a matrix of second-order statistics can be exploited to obtain a whitening matrix W ∈ R r×n such that the vectors

An analysis of an algorithm employing this technique coupled with tensor power iteration was carried out in #OTHEREFR , including a robust estimation result quantifying the performance in the case one observes Y = X + E, in terms of the spectral norm of the perturbation E.

Hotelling Deflation On Large Symmetric Spiked Tensors

Abstract: -Cet article étudie l'algorithme de déflation appliqué à l'estimation d'un modèle de rang faible symétrique contenu dans un tenseur de grandes dimensions corrompu par un bruit additif gaussien. Plus précisément, nous fournissons une caractérisation précise de la performance en grandes dimensions de la déflation en termes des alignements des vecteurs obtenus par approximations successives de rang 1 et de leurs poids, en supposant des corrélations (fixes) non-triviales entre les composantes du modèle. Notre analyse permet de comprendre le mécanisme de déflation en présence de bruit et peut être exploitée pour concevoir des méthodes d'estimation plus efficaces.

Subtracting A Best Rank-1 Approximation May Increase Tensor Rank

Abstract: It has been shown that a best rank-R approximation of an order-k tensor may not exist when R>1 and k>2. This poses a serious problem to data analysts using tensor decompositions. It has been observed numerically that, generally, this issue cannot be solved by consecutively computing and subtracting best rank-1 approximations. The reason for this is that subtracting a best rank-1 approximation generally does not decrease tensor rank. In this paper, we provide a mathematical treatment of this property for real-valued 2x2x2 tensors, with symmetric tensors as a special case. Regardless of the symmetry, we show that for generic 2x2x2 tensors (which have rank 2 or 3), subtracting a best rank-1 approximation results in a tensor that has rank 3 and lies on the boundary between the rank-2 and rank-3 sets. Hence, for a typical tensor of rank 2, subtracting a best rank-1 approximation increases the tensor rank.